Java process utilizing 100% CPU in RTC 3.0.1.2

Sofus Albertsen (47●10●14) | asked Sep 26 '12, 6:19 a.m.

Specs:
Running virtual guest OS Linux with 4 gb of ram, 4 cores, and enough disk space
I don't know for how long time this has been going on, but i have noticed that the Java process of
My Htop output looks like this.
I'm not that smart when it comes to administering the server, but i cannot see anything that resembles a looping faliure or the likes in the logs.
Are there anyone who's got a clue on what could be the cause?
-Sofus Albertsen

3 answers

Most liked answers ↑|Newest answers|Oldest answers

permanent link

Clement Liu (1.5k●5●42●49) | answered Sep 26 '12, 10:07 a.m.

Go to RTC admin console -> Active Services and check if there are any java threads running. If yes, what are they?

Comments

Sofus Albertsen commented Sep 26 '12, 10:19 a.m. | edited Sep 26 '12, 10:20 a.m.

com.ibm.team.repository.service.internal.ServerStatusRestService.getActiveServiceInfo

With this as the stacktrace i think:

java.lang.Thread.getStackTraceImpl(Native Method) java.lang.Thread.getStackTrace(Thread.java:1147) com.ibm.team.repository.service.internal.ServerStatusRestService.renderThreadStack(ServerStatusRestService.java:314) com.ibm.team.repository.service.internal.ServerStatusRestService.getActiveServiceInfo(ServerStatusRestService.java:304) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:618) org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370) org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356)

There's more but im not allowed to post that much. Hope i helps

Bo Chulindra commented Sep 26 '12, 12:05 p.m.

JAZZ DEVELOPER

@clementliu: I think this would be better as added as a comment to the original question. These comments of yours are posted as "answers" which is a bit confusing if someone is looking for answers but instead see your comments.

Clement Liu commented Sep 26 '12, 12:12 p.m.

@bkchulin, :) thanks. But looks like I can't upload screen shot to a comment so that's why sometimes I had to start a new conversation

Bo Chulindra commented Sep 26 '12, 12:27 p.m.

JAZZ DEVELOPER

@clementliu: ah, good point. I've posted the question https://jazz.net/forum/questions/89178/is-it-possible-to-add-a-screenshot-when-posting-a-forum-comment hoping that if it is not possible, then it may be made possible.

permanent link

Clement Liu (1.5k●5●42●49) | answered Sep 26 '12, 10:28 a.m.

There is running time at the end of the line of each thread... Can you paste those running for hours, if any?

Comments

Sofus Albertsen commented Sep 26 '12, 10:35 a.m.

00:00:00:000 running time so no, that's not it. if i do a ps -wwfC java i get the following: UID PID PPID C STIME TTY TIME CMD root 1529 1 99 11:18 ? 05:49:27 /opt/IBM/JazzTeamServer/server/jre/bin/java -Djava.util.logging.config.file=/opt/IBM/JazzTeamServer/server/tomcat/conf/logging.properties -Djazz.connector.sslProtocol=SSL_TLS -Djazz.connector.algorithm=IbmX509 -Djava.awt.headless=true -DSQLSERVER_JDBC=/opt/IBM/JazzTeamServer/server/sqlserver -DSQLSERVER_JDBC_DRIVER_FILE= -DORACLE_JDBC=/opt/IBM/JazzTeamServer/server/oracle -DORACLE_JDBC_DRIVER_FILE= -DDB2I_JDBC=/opt/IBM/JazzTeamServer/server/db2i -DDB2Z_JDBC=/opt/IBM/JazzTeamServer/server/db2z -Dorg.eclipse.emf.ecore.plugin.EcorePlugin.doNotLoadResourcesPlugin=true -DJAZZ_HOME=file:///opt/IBM/JazzTeamServer/server/conf -Dcom.ibm.team.repository.tempDir=/tmp -Xmx1200M -Dlog4j.configuration=file:///opt/IBM/JazzTeamServer/server/conf/startup_log4j.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/opt/IBM/JazzTeamServer/server/tomcat/common/endorsed -classpath /opt/IBM/JazzTeamServer/server/tomcat/bin/bootstrap.jar

Sofus Albertsen commented Sep 26 '12, 10:36 a.m.

And these 3 lines for the end of the process: -Dcatalina.base=/opt/IBM/JazzTeamServer/server/tomcat -Dcatalina.home=/opt/IBM/JazzTeamServer/server/tomcat -Djava.io.tmpdir=/opt/IBM/JazzTeamServer/server/tomcat/temp org.apache.catalina.startup.Bootstrap start

permanent link

Clement Liu (1.5k●5●42●49) | answered Sep 26 '12, 10:39 a.m.

Can you actually take a full screen shot of this in your environment?

Comments

Sofus Albertsen commented Sep 26 '12, 10:42 a.m.

https://dl.dropbox.com/u/697024/Screenshot%20from%202012-09-26%2016%3A41%3A19.png

Clement Liu commented Sep 26 '12, 10:49 a.m.

Hrm... getActiveServiceInfo is the only one? Then doesn't look like RTC server is busy with anything... wait... seems like you're looking at the JTS admin console... not the RTC admin console...

Can you actually take a screen shot of your RTC admin console?

The url should be:

https://hostname:port/ccm/admin

https://hostname:port/jazz/admin

Sofus Albertsen commented Sep 26 '12, 10:56 a.m.

Darn, i apologize for that one: Seems there acctually is another thread running

https://dl.dropbox.com/u/697024/rtc/Screenshot%20from%202012-09-26%2016%3A53%3A26.png https://dl.dropbox.com/u/697024/rtc/Screenshot%20from%202012-09-26%2016%3A53%3A33.png https://dl.dropbox.com/u/697024/rtc/Screenshot%20from%202012-09-26%2016%3A53%3A41.png

Clement Liu commented Sep 26 '12, 11:10 a.m.

That thread is actually OK, only running for 7 secs and I bet it will disappear soon. Hrm... it doesn't look either JTS or RTC is busy... From your screen shot of Htop, I am wondering why there are so many Java processes running at the same time and each grabbing 34% of Memory??? I would suggest to submit a PMR to IBM support for them to dig into...Sorry...not a great help..

Ralph Schoon commented Sep 26 '12, 11:36 a.m.

FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

Could this be a garbage collection issue?

Sofus Albertsen commented Sep 26 '12, 12:47 p.m.

@Clement, I will try that, but thanks for your help anyway! @Ralph, i have tried to restart the server, and right after the initial startup, the process is running. if i refresh the active services, the com.ibm.team.repository.service.internal.QueryService.queryItems seems to be running exactly 14 seconds, and the starts all over again. I suspect somekind of deadlock?

Sofus Albertsen commented Sep 27 '12, 3:51 a.m.

I have created a PMR here.
Hope it helps.

Krzysztof Kaźmierczyk commented Apr 11 '13, 4:13 a.m.

Hi Sofus,
Were you able to fix the issue? What was the root cause and the resolution?

showing 5 of 8 show 3 more comments

It's all about the answers!

Related questions

Java process utilizing 100% CPU in RTC 3.0.1.2

3 answers

Your answer