Tag Archives: e-business suite

AD Patch Worker Hangs on XDOLoader Process

Published by:

Have you run an e-Business Suite R12 patch that slowed down or hung at the Java Loader steps for no apparent reason? I first encountered this issue in January, and finding a workable solution took several hours of research. No Oracle Support notes pointed directly to the issue at the time, although several more recent notes make the issue easier to identify and solve. Hopefully this post will be useful to someone else.

Platform: Red Hat Enterprise Linux Server
Application Version: e-Business Suite 12.1+

Symptoms:

Patch runs fine until it begins to slow down and hang partway through the java loader (e.g., XDOLoader) steps for no apparent reason. There are no indications that the hang is being caused by a database performance or locking issue.

Troubleshooting:

AD patch worker log error:

Error: Error connecting to database "jdbc:oracle:thin:APPS/xxxxxx@(DESCRIPTION=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=YOUR_HOST)(PORT=1521)))(CONNECT_DATA=(SID=YOUR_SID)))"
Io exception: Connection reset

Run jstack on the hanging java process:

"main" prio=10 tid=0x08937000 nid=0x22ea runnable [0xf73e1000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:199)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0xf29b25a0> (a java.io.BufferedInputStream)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0xf29b2370> (a java.io.BufferedInputStream)
at sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte(SeedGenerator.java:453)
at sun.security.provider.SeedGenerator.getSeedBytes(SeedGenerator.java:123)
at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:118)
at sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:114)
at sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:171)
- locked <0xf29b1fd0> (a sun.security.provider.SecureRandom)
at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
- locked <0xf29b2250> (a java.security.SecureRandom)
at oracle.security.o5logon.O5LoginClientHelper.generateOAuthResponse(Unknown Source)
at oracle.jdbc.driver.T4CTTIoauthenticate.marshalOauth(T4CTTIoauthenticate.java:457)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:367)
at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:510)
at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:203)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at oracle.apps.xdo.oa.util.XDOLoader.initAppsContext(XDOLoader.java:558)
at oracle.apps.xdo.oa.util.XDOLoader.init(XDOLoader.java:455)
at oracle.apps.xdo.oa.util.XDOLoader.(XDOLoader.java:413)
at oracle.apps.xdo.oa.util.XDOLoader.main(XDOLoader.java:2250)

Check /dev/random entropy:

cat /proc/sys/kernel/random/entropy_avail
NOTE: Higher numbers are better. The patch will begin to slow down or hang whenever entropy is ~50 or less.

Explanation:

The java process depends on the /dev/random device to provide random numbers to the SecureRandom Java class. If /dev/random runs out of random numbers, the patch workers calling SecureRandom hang until enough random numbers are available.

Solutions:
NOTE: Pick one of the solutions below. Solution number 1 is my preferred solution, since it is specific to the e-Business Suite and should not affect other processes on the server.

  1. Search for all jre/lib/security/java.security files and replace:

    securerandom.source=file:/dev/random
    with
    securerandom.source=file:/dev/urandom

  2. Run the rngd daemon to seed /dev/random with random numbers:
    Install the rngd-utils package in RedHat 5 or kernel-utils in RedHat 4.
    rngd -r /dev/urandom -o /dev/random -f -t 1
  3. Replace the /dev/random device with /dev/urandom. (Not recommended for security reasons.)

    sudo mv /dev/random /dev/random.bak
    sudo ln -s /dev/urandom /dev/random

References:

Linux + Oracle Apps + Sun JRE Plugin

Published by:

I recently upgraded my production instance to 11i.ATG_PF.H RUP 5 and replaced JInitiator with the Sun JRE Plugin. To be able to use Linux with the application’s applets, just perform the following steps:

  1. Install Mozilla Firefox if you don’t already have it.
  2. Download the supported JRE. In my case, it is 1.5.0_12. Because Oracle cannot keep up with new technology, I had to dig around in the Sun Java Archive to find it.
  3. Copy the JRE bin file to where you want it installed. I copied mine to /opt.
  4. Make the bin file executable:
    sudo chmod 744 jre-1_5_0_12-linux-i586.bin
  5. Install the bin file:
    sudo jre-1_5_0_12-linux-i586.bin
  6. Create a symlink to the JRE plugin:
    sudo ln -s /opt/jre1.5.0_12/plugin/i386/ns7/libjavaplugin_oji.so /usr/lib/firefox/plugins
    UPDATE: For Firefox 3:
    sudo ln -s /opt/jre1.5.0_12/plugin/i386/ns7/libjavaplugin_oji.so /usr/lib/firefox-addons/plugins
  7. Startup Firefox, and give it a whirl.