Job server connection problems
RAM 7.5/WAS 7.0.0.11 on Win2003 server, SQL2008.
I've been going through the RAM logs, and the following entry is consistently appearing: 00000037 SystemOut O CRRAM0001E 4743551 ERROR ws com.ibm.ram.repository.web.ws.quartz.jobs.AssetIndexBuilderJob - Error communicating with job server java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:180) at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:452) at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:494) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:222) at java.io.InputStreamReader.read(InputStreamReader.java:177) at java.io.BufferedReader.fill(BufferedReader.java:148) at java.io.BufferedReader.read(BufferedReader.java:169) at com.ibm.ram.repository.web.ws.quartz.jobs.AssetIndexBuilderJob.executeJob(AssetIndexBuilderJob.java:59) at com.ibm.ram.repository.web.ws.quartz.jobs.AbstractRAMJob.execute(AbstractRAMJob.java:80) at org.quartz.core.JobRunShell.run(JobRunShell.java:203) at com.ibm.ram.repository.web.ws.was.quartz.RAMThreadPool$RAMWork.run(RAMThreadPool.java:166) at com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java:268) at java.security.AccessController.doPrivileged(AccessController.java:202) at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:768) at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:222) at com.ibm.ws.asynchbeans.ABWorkItemImpl.run(ABWorkItemImpl.java:159) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1550) 00000036 SystemOut O CRRAM0002W 4790034 WARN ws com.ibm.ram.repository.jobs.server.JobServerManager - Exception getting job server connection java.net.ConnectException: Connection refused: connect at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:352) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:214) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:201) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:378) at java.net.Socket.connect(Socket.java:537) at java.net.Socket.connect(Socket.java:487) at java.net.Socket.<init>(Socket.java:384) at java.net.Socket.<init>(Socket.java:227) at com.ibm.ram.repository.jobs.server.JobServerManager.getServerConnection(JobServerManager.java:311) at com.ibm.ram.repository.jobs.server.JobServerManager.getJobServerConnection(JobServerManager.java:366) at com.ibm.ram.repository.web.ws.quartz.jobs.ForumIndexBuilderJob.executeJob(ForumIndexBuilderJob.java:41) at com.ibm.ram.repository.web.ws.quartz.jobs.AbstractRAMJob.execute(AbstractRAMJob.java:80) at org.quartz.core.JobRunShell.run(JobRunShell.java:203) at com.ibm.ram.repository.web.ws.was.quartz.RAMThreadPool$RAMWork.run(RAMThreadPool.java:166) at com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java:268) at java.security.AccessController.doPrivileged(AccessController.java:202) at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:768) at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:222) at com.ibm.ws.asynchbeans.ABWorkItemImpl.run(ABWorkItemImpl.java:159) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1550) I've searched for references on this but not found anything, anyone have any ideas on why this is occurring? |
7 answers
By default, the indexer is working on a separate process (outside of the WAS VM).
When was the last time your indexer ran ? (check this from the Administration -> Tools -> Job Status) |
Hi,
1) Look in the indexJobServerSystemOutLog.txt and indexJobServerSystemErrLog.txt logs to verify that your index jobserver is even starting, or if there are errors starting them. 2) Look in the ramJobServerDebug.log to verify that there were no errors running the job process that would of caused it to go away. 3) Verify that you don't have a firewall running on your system that is blocking RAM on that machine from accessing the port that the index job process uses to communicate. By default RAM uses the port range 4545 to 4555 to communicate between the app server instance and the index job process. Rich |
By default, the indexer is working on a separate process (outside of the WAS VM). This is from the Job Status: Asset Index Builder No update necessary May 5, 2011 3:58:25 PM May 5, 2011 3:58:25 PM 0 seconds |
Hi, From the indexJobServerSystemErrLog.txt, it is full of: Exception in thread "Thread-16217" com.ibm.ram.repository.core.IRepositoryAccess$RepositoryException: The TCP/IP connection to the host SQLAU182DEV115, port 50270 has failed. Error: "connect timed out. Verify the connection properties, check that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port, and that no firewall is blocking TCP connections to the port.". This log has no time stamps, so it is not possible to tell when these errors are occurring. There is nothing wrong with the DB connection parameters, or the DB itself. There are no firewalls in place. From the ramJobServerDebug.log: CRRAM0001E 27607748 ERROR com.ibm.ram.repository.fsearch.assets.AssetIndexBuilder - Error indexing reviewer data for asset 2048 java.lang.NoClassDefFoundError: com.ibm.ram.client.RAMCommunity at java.lang.J9VMInternals.verifyImpl(Native Method) at java.lang.J9VMInternals.verify(J9VMInternals.java:72) at java.lang.J9VMInternals.initialize(J9VMInternals.java:134) at com.anz.emap.ram.governor.EmapGovernor.initPolicies(EmapGovernor.java:77) at com.anz.emap.ram.governor.EmapGovernor.getPolicies(EmapGovernor.java:60) at com.anz.emap.ram.governor.EmapGovernor.getAssetPolicies(EmapGovernor.java:26) at com.ibm.ram.repository.rest.RestUtils.getPolicyGovernor(RestUtils.java:1733) at com.ibm.ram.repository.rest.RestPoliciesService.doGet(RestPoliciesService.java:161) at com.ibm.ram.repository.lifecycle.AssetLifecycleManager.getPolicyDefinition(AssetLifecycleManager.java:4226) at com.ibm.ram.repository.lifecycle.AssetLifecycleManager.getPopulatedStateHistory(AssetLifecycleManager.java:1187) at com.ibm.ram.repository.lifecycle.AssetLifecycleManager.getActiveStateHistory(AssetLifecycleManager.java:1486) at com.ibm.ram.repository.lifecycle.AssetLifecycleManager.getActiveStateHistory(AssetLifecycleManager.java:1456) at com.ibm.ram.repository.review.Utilities.getRequiredReviewers(Utilities.java:2166) at com.ibm.ram.repository.fsearch.assets.AssetIndexBuilder.createLuceneDoc(AssetIndexBuilder.java:917) at com.ibm.ram.repository.fsearch.assets.AssetIndexBuilder.updateDocs(AssetIndexBuilder.java:255) at com.ibm.fsearch.index.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:188) at com.ibm.ram.repository.fsearch.assets.AssetIndexBuilder.buildRepositoryIndex(AssetIndexBuilder.java:1493) at com.ibm.ram.repository.jobs.AssetIndexJob.run(AssetIndexJob.java:222) at java.lang.Thread.run(Thread.java:736) Caused by: java.lang.ClassNotFoundException: com.ibm.ram.client.RAMCommunity at java.lang.ClassNotFoundException.<init>(ClassNotFoundException.java:77) at java.net.URLClassLoader.findClass(URLClassLoader.java:383) at java.lang.ClassLoader.loadClass(ClassLoader.java:652) at java.lang.ClassLoader.loadClass(ClassLoader.java:618) ... 19 more |
From the ramJobServerDebug.log: It seems that you have a custom policy governor (com.anz.emap.ram.governor.EmapGovernor) That uses the ram client api, but does not have the ramclient.jar on its path. either add it to the WEB-INF/lib directory for the com.ibm.repository.web.war and com.ibm.repository.web.ws.was.war ... or .zip it with you policy .jar and add it as you policy. |
From the ramJobServerDebug.log: The above error was resolved in RAM 7.5.0.1. See https://jazz.net/jazz02/resource/itemName/com.ibm.team.workitem.WorkItem/41033 If possible, you should upgrade to RAM 7.5.0.2. As a workaround in RAM 7.5, you will have to navigate to http://<host>/ram/admin/repository/advancedConfiguration.faces and uncheck the 'Perform all indexing in a remote process' checkbox. This will force the indexing process in run in the WAS java process. The indexer normally runs on a low priority thread at the OS level. When running in the WAS java process it will run in a low priority thread at the JVM level. This means the indexer may use more CPU and you may see reduced performance in other areas of RAM if you do not have enough CPU power available. Adding the JARs to the WEB-INF/lib folder as stated in the above comment will not resolve the issue you are seeing.
This error says that the index process was unable to make a database connection. I'm not sure what was causing the error, but if you hit the first error it implies that the indexer eventually made the DB connection and started successfully. |
|
Regarding the class not found (for the RichClient API) ... it is likely that the ramclient.jar is missing still (for the policies on that lifecycle)...
make sure you have that .jar packaged as part of the policy or in WEB-INF. |
Your answer
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.