Job server connection problems

By default, the indexer is working on a separate process (outside of the WAS VM).


When was the last time your indexer ran ? (check this from the Administration -> Tools -> Job Status)

Hi,

1) Look in the indexJobServerSystemOutLog.txt and
indexJobServerSystemErrLog.txt logs to verify that your index jobserver
is even starting, or if there are errors starting them.

2) Look in the ramJobServerDebug.log to verify that there were no errors
running the job process that would of caused it to go away.

3) Verify that you don't have a firewall running on your system that is
blocking RAM on that machine from accessing the port that the index job
process uses to communicate. By default RAM uses the port range 4545 to
4555 to communicate between the app server instance and the index job
process.

Rich

By default, the indexer is working on a separate process (outside of the WAS VM).


When was the last time your indexer ran ? (check this from the Administration -> Tools -> Job Status)

This is from the Job Status:

Asset Index Builder No update necessary May 5, 2011 3:58:25 PM May 5, 2011 3:58:25 PM 0 seconds

Hi,

1) Look in the indexJobServerSystemOutLog.txt and
indexJobServerSystemErrLog.txt logs to verify that your index jobserver
is even starting, or if there are errors starting them.

2) Look in the ramJobServerDebug.log to verify that there were no errors
running the job process that would of caused it to go away.

3) Verify that you don't have a firewall running on your system that is
blocking RAM on that machine from accessing the port that the index job
process uses to communicate. By default RAM uses the port range 4545 to
4555 to communicate between the app server instance and the index job
process.

Rich

From the indexJobServerSystemErrLog.txt, it is full of:

Exception in thread "Thread-16217" com.ibm.ram.repository.core.IRepositoryAccess$RepositoryException: The TCP/IP connection to the host SQLAU182DEV115, port 50270 has failed. Error: "connect timed out. Verify the connection properties, check that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port, and that no firewall is blocking TCP connections to the port.".

This log has no time stamps, so it is not possible to tell when these errors are occurring. There is nothing wrong with the DB connection parameters, or the DB itself.

There are no firewalls in place.

From the ramJobServerDebug.log:

CRRAM0001E 27607748 ERROR com.ibm.ram.repository.fsearch.assets.AssetIndexBuilder - Error indexing reviewer data for asset 2048
java.lang.NoClassDefFoundError: com.ibm.ram.client.RAMCommunity
at java.lang.J9VMInternals.verifyImpl(Native Method)
at java.lang.J9VMInternals.verify(J9VMInternals.java:72)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:134)
at com.anz.emap.ram.governor.EmapGovernor.initPolicies(EmapGovernor.java:77)
at com.anz.emap.ram.governor.EmapGovernor.getPolicies(EmapGovernor.java:60)
at com.anz.emap.ram.governor.EmapGovernor.getAssetPolicies(EmapGovernor.java:26)
at com.ibm.ram.repository.rest.RestUtils.getPolicyGovernor(RestUtils.java:1733)
at com.ibm.ram.repository.rest.RestPoliciesService.doGet(RestPoliciesService.java:161)
at com.ibm.ram.repository.lifecycle.AssetLifecycleManager.getPolicyDefinition(AssetLifecycleManager.java:4226)
at com.ibm.ram.repository.lifecycle.AssetLifecycleManager.getPopulatedStateHistory(AssetLifecycleManager.java:1187)
at com.ibm.ram.repository.lifecycle.AssetLifecycleManager.getActiveStateHistory(AssetLifecycleManager.java:1486)
at com.ibm.ram.repository.lifecycle.AssetLifecycleManager.getActiveStateHistory(AssetLifecycleManager.java:1456)
at com.ibm.ram.repository.review.Utilities.getRequiredReviewers(Utilities.java:2166)
at com.ibm.ram.repository.fsearch.assets.AssetIndexBuilder.createLuceneDoc(AssetIndexBuilder.java:917)
at com.ibm.ram.repository.fsearch.assets.AssetIndexBuilder.updateDocs(AssetIndexBuilder.java:255)
at com.ibm.fsearch.index.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:188)
at com.ibm.ram.repository.fsearch.assets.AssetIndexBuilder.buildRepositoryIndex(AssetIndexBuilder.java:1493)
at com.ibm.ram.repository.jobs.AssetIndexJob.run(AssetIndexJob.java:222)
at java.lang.Thread.run(Thread.java:736)
Caused by:
java.lang.ClassNotFoundException: com.ibm.ram.client.RAMCommunity
at java.lang.ClassNotFoundException.<init>(ClassNotFoundException.java:77)
at java.net.URLClassLoader.findClass(URLClassLoader.java:383)
at java.lang.ClassLoader.loadClass(ClassLoader.java:652)
at java.lang.ClassLoader.loadClass(ClassLoader.java:618)
... 19 more

From the ramJobServerDebug.log:

CRRAM0001E 27607748 ERROR com.ibm.ram.repository.fsearch.assets.AssetIndexBuilder - Error indexing reviewer data for asset 2048
java.lang.NoClassDefFoundError: com.ibm.ram.client.RAMCommunity
at java.lang.J9VMInternals.verifyImpl(Native Method)

It seems that you have a custom policy governor (com.anz.emap.ram.governor.EmapGovernor) That uses the ram client api, but does not have the ramclient.jar on its path.

either add it to the WEB-INF/lib directory for the com.ibm.repository.web.war and com.ibm.repository.web.ws.was.war ... or .zip it with you policy .jar and add it as you policy.

From the ramJobServerDebug.log:

CRRAM0001E 27607748 ERROR com.ibm.ram.repository.fsearch.assets.AssetIndexBuilder - Error indexing reviewer data for asset 2048
java.lang.NoClassDefFoundError: com.ibm.ram.client.RAMCommunity
at java.lang.J9VMInternals.verifyImpl(Native Method)

The above error was resolved in RAM 7.5.0.1. See
https://jazz.net/jazz02/resource/itemName/com.ibm.team.workitem.WorkItem/41033
If possible, you should upgrade to RAM 7.5.0.2.

As a workaround in RAM 7.5, you will have to navigate to
http://<host>/ram/admin/repository/advancedConfiguration.faces
and uncheck the 'Perform all indexing in a remote process' checkbox. This will force the indexing process in run in the WAS java process. The indexer normally runs on a low priority thread at the OS level. When running in the WAS java process it will run in a low priority thread at the JVM level. This means the indexer may use more CPU and you may see reduced performance in other areas of RAM if you do not have enough CPU power available.

Adding the JARs to the WEB-INF/lib folder as stated in the above comment will not resolve the issue you are seeing.

Exception in thread "Thread-16217" com.ibm.ram.repository.core.IRepositoryAccess$RepositoryException: The TCP/IP connection to the host SQLAU182DEV115, port 50270 has failed. Error: "connect timed out. Verify the connection properties, check that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port, and that no firewall is blocking TCP connections to the port.".

This error says that the index process was unable to make a database connection. I'm not sure what was causing the error, but if you hit the first error it implies that the indexer eventually made the DB connection and started successfully.

Regarding the class not found (for the RichClient API) ... it is likely that the ramclient.jar is missing still (for the policies on that lifecycle)...

make sure you have that .jar packaged as part of the policy or in WEB-INF.