My CLM applications are crashing again and again after every 6-7 hours.Have to restart the jts services to make applications up and running
the applications RTC, RRC and RQM are crashing after every 6-7 hours and need to restart the jts services again and again. We are using Apache tomcat as Application server
Accepted answer
2 other answers
Hi Pratibha,
I found your PMR and the javacores
It looks like you have 3 java processes running each using about 5 gigs of memory.
That is cutting it tight since it looks like your system has 16 gigs total memory.
One thing that might help the hangs would be setting the XMS value in the server.startup file
I am a bit concerned about having all three applications set to use 4 gigs of memory each as we don't want you to run out of system memory.
A resolution might require a bit of trial and error to get it right (unless you want to add system memory)
I would consider setting the QM and RM applications to use less, maybe set them to 3 gigs each and leave CCM at 4 gigs. Provided that CCM is the application that is under the heaviest use.
You would want to keep an eye on general performance after making that change.
Setting the XMS value:
It should be set to the same value as the XMX setting.
server.startup will be in the following location:
/opt/IBM/JazzTeamServer/server
Currently you have:
set JAVA_OPTS=%JAVA_OPTS% -Xmx4000M
You will want to add this line after that one:
set JAVA_OPTS=%JAVA_OPTS% -Xms4000M
i.e.
set JAVA_OPTS=%JAVA_OPTS% -Xmx4000M
set JAVA_OPTS=%JAVA_OPTS% -Xms4000M
It will require a restart (or you can just wait till you are forced to restart)
One other thing I noticed in the PMR is that you restarted the database at some point and had trouble with RTC afterwards. If you restart the database it is advisable to restart RTC as well because there are usually stale connections to the database that need to be cleaned up.
I found your PMR and the javacores
It looks like you have 3 java processes running each using about 5 gigs of memory.
That is cutting it tight since it looks like your system has 16 gigs total memory.
One thing that might help the hangs would be setting the XMS value in the server.startup file
I am a bit concerned about having all three applications set to use 4 gigs of memory each as we don't want you to run out of system memory.
A resolution might require a bit of trial and error to get it right (unless you want to add system memory)
I would consider setting the QM and RM applications to use less, maybe set them to 3 gigs each and leave CCM at 4 gigs. Provided that CCM is the application that is under the heaviest use.
You would want to keep an eye on general performance after making that change.
Setting the XMS value:
It should be set to the same value as the XMX setting.
server.startup will be in the following location:
/opt/IBM/JazzTeamServer/server
Currently you have:
set JAVA_OPTS=%JAVA_OPTS% -Xmx4000M
You will want to add this line after that one:
set JAVA_OPTS=%JAVA_OPTS% -Xms4000M
i.e.
set JAVA_OPTS=%JAVA_OPTS% -Xmx4000M
set JAVA_OPTS=%JAVA_OPTS% -Xms4000M
It will require a restart (or you can just wait till you are forced to restart)
One other thing I noticed in the PMR is that you restarted the database at some point and had trouble with RTC afterwards. If you restart the database it is advisable to restart RTC as well because there are usually stale connections to the database that need to be cleaned up.
Comments
Rajat Singh
May 07 '13, 10:01 a.m.Hi Pratibha
Matt Lennon
JAZZ DEVELOPER May 09 '13, 2:36 a.m.Hi Pratibha.
You should also check your CLM server logs
<clm-server-install-dir/server/logs/{ccm,jts,qm}.log
and Tomcat logs for log entries corresponding to the time of the crashes.
What version of CLM are you running?
-Matt
pratibha gupta
May 02 '13, 1:29 a.m.We are not able to log into the applications and have to restart the jts server . We are using RHEL 5.0 OS
pratibha gupta
May 02 '13, 1:30 a.m.I am using CLM 3.0.1.
Karl Weinert
JAZZ DEVELOPER May 09 '13, 2:35 a.m.How much memory do you have on your system?
Also check for javacore files somewhere in the path leading up to the directory where you are running java from.
I have not found a consistent place where they get created but here's a couple places to check:
/opt/IBM/JazzTeamServer/server
/opt/IBM/JazzTeamServer/server/jre/bin
They may even be in the tomcat directory structure.
/opt/IBM/JazzTeamServer/server/tomcat
/opt/IBM/JazzTeamServer/server/tomcat/bin
Or just search /opt/IBM/JazzTeamServer for "javacore*.txt"
If you find some that correspond to the time of the crashes the first 10 lines or so should provide more details on what caused the javacores to be created. Feel free to paste them in.
pratibha gupta
May 07 '13, 9:00 a.m.I checked on all the paths mentioned by you but could not find javacore*.txt files. Please let me know if there is any other path
Karl Weinert
JAZZ DEVELOPER May 07 '13, 9:23 a.m.Hi Pratibha,
This is probably getting a little complex to troubleshoot via the forum.
It would be good to know how much memory your system has and how much of it is dedicated to the jvm
I would suggest opening a PMR but if you can't I would start by collecting WAIT tool output run for a period of time leading up to the time the system hangs. Start by running it every 15-30 minutes for an hour or two before the problem usually occurs.
This will output quite a bit of performance data that might help in figuring out what is going on.
You can download and read about the tool at https://wait.ibm.com/
I generally start looking at the javacores created by the wait tool with "IBM Thread and Monitor Dump Analyzer for Java" found here:
https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=2245aa39-fa5c-4475-b891-14c205f7333c
1 vote
pratibha gupta
May 07 '13, 9:26 a.m.I got javacore*.txt but not able to understand these file. Plz check
0SECTION ENVINFO subcomponent dump routine
NULL =================================
1CIJAVAVERSION J2RE 5.0 IBM J9 2.3 Linux amd64-64 build j9vmxa6423ifx-20100510
1CIVMVERSION VM build 20100509_57823_LHdSMr
1CIJITVERSION JIT enabled - 20091016_1845ifx7_r8
1CIRUNNINGAS Running as a standalone JVM
1CICMDLINE /opt/ibm/JazzTeamServer/server/jre/bin/java -Djava.util.logging.config.file=/opt/ibm/JazzTeamServer/server/tomcat/conf/logging.properties -Djazz.connector.sslProtocol=SSL_TLS -Djazz.connector.algorithm=IbmX509 -Djava.awt.headless=true -DSQLSERVER_JDBC=/opt/ibm/JazzTeamServer/server/sqlserver -DSQLSERVER_JDBC_DRIVER_FILE= -DORACLE_JDBC=/opt/ibm/JazzTeamServer/server/oracle -DORACLE_JDBC_DRIVER_FILE=/opt/ibm/JazzTeamServer/server/oracle/ojdbc5.jar -DDB2I_JDBC=/opt/ibm/JazzTeamServer/server/db2i -DDB2Z_JDBC=/opt/ibm/JazzTeamServer/server/db2z -Dorg.eclipse.emf.ecore.plugin.EcorePlugin.doNotLoad