Hung thread on com.ibm.team.tpt.internal.service.ImpexService.service?

Scott Crouch

(485●3●24●26) Apr 04 '13, 10:57 a.m.

We're occasionally seeing hung thread alerts from WebSphere, when we look in the active services of CCM we see active services with long running times for com.ibm.team.tpt.internal.service.ImpexService.service.

Has anybody seen this before? Any ideas what could be triggering it?

The hung thread stack trace appears to be:

[4/4/13 9:40:23:589 CDT] 00000046 ThreadMonitor W WSVR0605W: Thread "WebContainer : 8922" (0000be8e) has been active for 1495600 milliseconds and may be hung. There is/are 2 thread(s) in total in the server that may be hung.

at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.addNode(Unknown Source)

at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.nextNode(Unknown Source)

at org.apache.xml.dtm.ref.DTMDefaultBase._nextsib(Unknown Source)

at org.apache.xml.dtm.ref.DTMDefaultBase.getNextSibling(Unknown Source)

at org.apache.xml.dtm.ref.DTMDefaultBaseTraversers$ChildTraverser.next(Unknown Source)

at org.apache.xpath.axes.AxesWalker.getNextNode(Unknown Source)

at org.apache.xpath.axes.AxesWalker.nextNode(Unknown Source)

at org.apache.xpath.axes.WalkingIterator.nextNode(Unknown Source)

at org.apache.xpath.axes.NodeSequence.nextNode(Unknown Source)

at org.apache.xpath.axes.NodeSequence.runTo(Unknown Source)

at org.apache.xml.dtm.ref.DTMNodeList.<init>(Unknown Source)

at org.apache.xpath.objects.XNodeSet.nodelist(Unknown Source)

at org.apache.xpath.jaxp.XPathExpressionImpl.xObjectToObject(Unknown Source)

at org.apache.xpath.jaxp.XPathExpressionImpl.evaluate(Unknown Source)

at com.ibm.team.tpt.internal.service.util.ImpexUtil.retrieveDataList(ImpexUtil.java:608)

at com.ibm.team.tpt.internal.service.impex.AbstractXMLExportService.generateTaskInfo(AbstractXMLExportService.java:200)

at com.ibm.team.tpt.internal.service.impex.AbstractXMLExportService.generateTaskInfo(AbstractXMLExportService.java:312)

at com.ibm.team.tpt.internal.service.impex.AbstractXMLExportService.generateTasksInfo(AbstractXMLExportService.java:422)

at com.ibm.team.tpt.internal.service.impex.AbstractXMLExportService.generateDocumentFromObjects(AbstractXMLExportService.java:88)

at com.ibm.team.tpt.internal.service.impex.MSPXMLExportService.doExport(MSPXMLExportService.java:38)

at com.ibm.team.tpt.internal.service.ImpexService.perform_GET(ImpexService.java:89)

at com.ibm.team.repository.service.TeamRawService.service(TeamRawService.java:82)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

at java.lang.reflect.Method.invoke(Method.java:611)

at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)

at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)

at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)

at $Proxy1116.service(Unknown Source)

at com.ibm.team.repository.servlet.AbstractTeamServerServlet.doRestService(AbstractTeamServerServlet.java:914)

at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest2(AbstractTeamServerServlet.java:2100)

at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest(AbstractTeamServerServlet.java:1924)

at com.ibm.team.repository.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:1800)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)

WAS 7, RHEL, RTC 4.0.1.

0 votes

Comments

Scott Crouch

Apr 04 '13, 11:22 a.m.

This was also tied to high CPU usage for the web server. We were able to track down from HTTP logs that the user was likely trying to either export or import a plan from the Web UI. Still not sure what would trigger the problem. Can open a defect/PMR if that's the only answer, just wondering if anybody had seen this before or knew of an existing defect.

2 answers

10,030 views

0 votes

Accepted answer

Permanent link

Eric Jodet

JAZZ DEVELOPER (6.3k●5●111●120) Apr 04 '13, 11:37 a.m.

looking at the stack:

at org.apache.xpath.jaxp.XPathExpressionImpl.evaluate(Unknown Source)

at com.ibm.team.tpt.internal.service.util.ImpexUtil.retrieveDataList(ImpexUtil.java:608)

we seem to have problems to evaluate some xPath expression.

Not expected - which qualifies for a PMR - or a public work item if non confidential information can be shared

thecorresponding RTC log should here - and we could enable some tracing:

1 - stop the RTC server
2 - rotate the CCM log - rename ccm.log ccm.old.log
3 - edit server/conf/ccm/log4j.properties
4 - add this line at the bottom of the file:
log4j.logger.com.ibm.team.tpt.internal.service.ImpexService=DEBUG
5 - save your changes
6 - start the server
7 - do the import and wait for the issue to show up.
--> collect resulting ccm.log

Thanks

Scott Crouch selected this answer as the correct answer

1 vote

Comments

Krzysztof Kaźmierczyk

Apr 05 '13, 2:37 a.m.

Hi Eric, Do you think that it could be also useful also to provide MS Project file to try recreating it on site?

Eric Jodet

JAZZ DEVELOPER Apr 05 '13, 3:36 a.m.

Hello Krzys, well - it depends if the issue is with exporting from RTC to XML / MS Project or importing from MS Project back into RTC. For the later, yes, having the MS Project file would help a lot. Thanks. Eric.

Scott Crouch

Apr 05 '13, 8:43 a.m.

According to the user it seems they were trying to export from a plan, so there was no Microsoft project file involved. We haven't been able to reproduce the issue with the same plan. Since this is our production environment we'll have to get the change scheduled to do the bounce and update the log4j config. Would it be helpful to go ahead and open a jazz.net defect even though I may not be able to get any more DEBUG logging? This seems like a big deal because what we found is for a period of time we were spiking 3 of our CPUs (1 per hung thread), fortunately we had capacity to handle it, but it could have been worse.

Eric Jodet

JAZZ DEVELOPER Apr 05 '13, 9:05 a.m.

Hello Scott, unless we have a specific scenario to reproduce the issue in a systematic manner, I would say it is not useful to open a new defect or a PMR. Based on the fact we can not investigate / debug a case that we can not reproduce. On the other hand, you may update the log4 and trace the import / export. and try and come up with steps to reproduce the issue. Next time we have a similar issue, we may have more accurate information. Thanks. Eric.

Krzysztof Kaźmierczyk

Apr 08 '13, 3:48 a.m.

Hello Scott and Eric, Scott - I believe you can try openning the PMR anyway even if you do not have case to reproduce. RTC support is quite smart and might help you in narrowing down the issue and in discovering any patterns. Do you have maybye any other error messages in the log? Also how often does this error appear you in the log? Do you see any patterns here?

One other answer

Permanent link

Eric Jodet

JAZZ DEVELOPER (6.3k●5●111●120) Apr 04 '13, 11:22 a.m.

Hello Scott,

MSPXMLExportService is the code that enables the export of RTC Plans to MS Project / XML format.

If the thread is stuck, I would think that there is something preventing the operation to complete.

You may want to check with the user(s) performing this kind of operation.

If the problem persists, I would open a PMR - https://jazz.net/help/support/

Hope it helps.

Eric.

0 votes

Your answer

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here.

By RSS:

Answers

Answers and Comments

Question details

rational-team-concert

× 43,075

troubleshooting

× 6,081

jazz-foundation

× 4,381

Question asked: Apr 04 '13, 10:57 a.m.

Question was seen: 10,030 times

Last updated: Apr 08 '13, 3:48 a.m.

Hung thread on com.ibm.team.tpt.internal.service.ImpexService.service?

Comments

2 answers

10,030 views

0 votes

Accepted answer

Comments

One other answer

Your answer

Follow this question

Question details

Related questions