JTS server hanging, out of memory error
I have 4.0.0.1 JTS/CCM server with WAS8 - max heap is 4096. Periodically, I'm seeing the server hang with the following stack in the log. Is this a known problem? I'm thinking of setting min and max to 4096 an GC per this article https://jazz.net/forum/questions/78695/rtc-301-jvm-memory-sizing. Please let me know if this version 3.0.x setting is okay to use for 4.0.0.1 server.
com.ibm.team.repository.common.TeamRepositoryException: Error 500: java.lang.OutOfMemoryError: Failed to allocate JNIEnv at com.ibm.team.process.internal.authoring.service.ResourceCleanupService.deletedUnclaimedResourcesWithCancellation(ResourceCleanupService.java:198) at sun.reflect.GeneratedMethodAccessor548.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361) at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347) at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56) at $Proxy601.deletedUnclaimedResourcesWithCancellation(Unknown Source) at com.ibm.team.process.internal.authoring.service.ResourceCleanupTask.deleteUnclaimedProcessDescriptions(ResourceCleanupTask.java:153) at com.ibm.team.process.internal.authoring.service.ResourceCleanupTask.access$0(ResourceCleanupTask.java:145) at com.ibm.team.process.internal.authoring.service.ResourceCleanupTask$1.run(ResourceCleanupTask.java:64) at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42) at com.ibm.team.process.internal.authoring.service.ResourceCleanupTask.runTask(ResourceCleanupTask.java:61) at com.ibm.team.repository.service.async.AbstractAutoScheduledTask.executeTask(AbstractAutoScheduledTask.java:89) at sun.reflect.GeneratedMethodAccessor247.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361) at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347) at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56) at $Proxy602.executeTask(Unknown Source) at com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner.runTask(AsynchronousTaskRunner.java:136) at com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner.run(AsynchronousTaskRunner.java:99) at java.lang.Thread.run(Thread.java:737) Caused by: com.ibm.team.process.internal.authoring.common.utils.ProcessAuthoringException: Error 500: java.lang.OutOfMemoryError: Failed to allocate JNIEnv at com.ibm.team.process.internal.authoring.service.utils.QueryHelper.getProcessDescriptionsWithoutOwner(QueryHelper.java:3587) at com.ibm.team.process.internal.authoring.service.ResourceCleanupService.deletedUnclaimedResourcesWithCancellation(ResourceCleanupService.java:170) ... 23 more 2012-12-12 17:39:44,143 [ jts: AsynchronousTaskRunner-2] ERROR ess.internal.authoring.service.ResourceCleanupTask - com.ibm.team.repository.common.TeamRepositoryException: Error 500: java.lang.OutOfMemoryError: Failed to allocate JNIEnv |
4 answers
Hi Jim.
If you're server is experiencing out of memory exceptions, or even if you're running near the heap limit, you should bump up the heap size. Aside from getting OOM exceptions, JVM performance can really drop off if it's constantly having to run the garbage collector. You're at 4 GB now - try 6 GB to see if that takes care of the problem. As Rajat says, make sure you have enough RAM to avoid swapping. A good rule of thumb is *at least* 50% more RAM than the JVM heap max, plus whatever your other applications on that system need. Try these JVM memory settings: -Xmx6g -Xms6g -Xgcpolicy:gencon -Xcompressedrefs Setting "Xms" equal to "Xmx" avoids some unnecessary initialization and garbage collection. "-Xcompressedrefs" saves heap space and improves performance on 64-bit systems. "Xgcpolicy:gencon" is the garbage collection algorithm we've found to be most effective for normal RTC server work loads. You can get a rough idea of your current JVM heap usage on the JTS Server Admin "Status Summary" page. There are all sorts of tools out there for digging into heap usage in more detail. See Jazz.net work item 198140 for details about RTC JVM memory settings. -Matt Comments I have increased the JVM now, according your suggestion and also put more physical memory into the server. It looks quite well now, but we don't have the heavy load yet, because of holiday. My current setting on a Windows 2008-R2/64 bit machine with 24GB Ram
Will be interesting after new year holiday, when we got 100+ concurrent users on the system how it works.
I'm pursuing issues with SSL and having just done a gc analysis on a couple of the CLM applications that most often exhibit the SSL Handshake Exception, I can anectodally see that there is a possible association of the SSL handshake failure with bursts of gc activity ( e.g. lots of 1s collections but within a minute or two ). Reading about GC stuff I found two articles that discuss the gc policies available in WebSphere 8. Was the "balanced" policy examined at all ? in some of the studies to settle upon "gencon" ?
|
Even if you set the heap size as 4096, do you have enough memory on the machine to allocate this?
What is the ram size on this machine? Is it hosting other applications in parallel? (db server, application server,any other application that consumes more memory) |
We have exactly the sdame problem, since upgrading to V.4.0.0.1. We run V.3.0.1.x since 14 Month without any problem. After upgrading to V.4.0.0.1 we have regularly after 12-24 hours an out-ofmemory exception in CCM. JTS is the using near 100% CPU on all cores of the machine. Only a restart of the WAS service helps. to survive the next few hours. We have 16GB Ram, JVM set to 4096. We run an All-In-1 machine with all CLM applications in one WAS profile plus DB2 server, all on same machine. I know you will say, distribute it to multiple machines, add more physical memory etc.. But this is still not an explanation why the application has problems since the 4.0.0.1 upgrade. I'm absolutly sure, there is a memory leak in one of the application. I think in CCM. We will move the whole stuff to a new hardware within next few weeks. With 128GB Ram. And we will distribute all applications into own profile with 4GB JVM each. Hope it is then enough for Jazz.
|
|
We are using Vm image which has been allocated 8g RAM, 4 processors using REL6 running JTS/CCM/WebSphere only. There are no other Vms on this physical machine which is an IBM 3350 and has 64g RAM.
Also, I only have 2-10 users on the system simultaneously at any given time with very low usage. |
Your answer
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.