Hung thread on com.ibm.team.tpt.internal.service.ImpexService.service?
We're occasionally seeing hung thread alerts from WebSphere, when we look in the active services of CCM we see active services with long running times for com.ibm.team.tpt.internal.service.ImpexService.service.
Has anybody seen this before? Any ideas what could be triggering it?
The hung thread stack trace appears to be:
[4/4/13 9:40:23:589 CDT] 00000046 ThreadMonitor W WSVR0605W: Thread "WebContainer : 8922" (0000be8e) has been active for 1495600 milliseconds and may be hung. There is/are 2 thread(s) in total in the server that may be hung.
at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.addNode(Unknown Source)
at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.nextNode(Unknown Source)
at org.apache.xml.dtm.ref.DTMDefaultBase._nextsib(Unknown Source)
at org.apache.xml.dtm.ref.DTMDefaultBase.getNextSibling(Unknown Source)
at org.apache.xml.dtm.ref.DTMDefaultBaseTraversers$ChildTraverser.next(Unknown Source)
at org.apache.xpath.axes.AxesWalker.getNextNode(Unknown Source)
at org.apache.xpath.axes.AxesWalker.nextNode(Unknown Source)
at org.apache.xpath.axes.WalkingIterator.nextNode(Unknown Source)
at org.apache.xpath.axes.NodeSequence.nextNode(Unknown Source)
at org.apache.xpath.axes.NodeSequence.runTo(Unknown Source)
at org.apache.xml.dtm.ref.DTMNodeList.<init>(Unknown Source)
at org.apache.xpath.objects.XNodeSet.nodelist(Unknown Source)
at org.apache.xpath.jaxp.XPathExpressionImpl.xObjectToObject(Unknown Source)
at org.apache.xpath.jaxp.XPathExpressionImpl.evaluate(Unknown Source)
at com.ibm.team.tpt.internal.service.util.ImpexUtil.retrieveDataList(ImpexUtil.java:608)
at com.ibm.team.tpt.internal.service.impex.AbstractXMLExportService.generateTaskInfo(AbstractXMLExportService.java:200)
at com.ibm.team.tpt.internal.service.impex.AbstractXMLExportService.generateTaskInfo(AbstractXMLExportService.java:312)
at com.ibm.team.tpt.internal.service.impex.AbstractXMLExportService.generateTasksInfo(AbstractXMLExportService.java:422)
at com.ibm.team.tpt.internal.service.impex.AbstractXMLExportService.generateDocumentFromObjects(AbstractXMLExportService.java:88)
at com.ibm.team.tpt.internal.service.impex.MSPXMLExportService.doExport(MSPXMLExportService.java:38)
at com.ibm.team.tpt.internal.service.ImpexService.perform_GET(ImpexService.java:89)
at com.ibm.team.repository.service.TeamRawService.service(TeamRawService.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at $Proxy1116.service(Unknown Source)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.doRestService(AbstractTeamServerServlet.java:914)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest2(AbstractTeamServerServlet.java:2100)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest(AbstractTeamServerServlet.java:1924)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:1800)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
WAS 7, RHEL, RTC 4.0.1.
Accepted answer
looking at the stack:
at org.apache.xpath.jaxp.XPathExpressionImpl.evaluate(Unknown Source)
at com.ibm.team.tpt.internal.service.util.ImpexUtil.retrieveDataList(ImpexUtil.java:608)
we seem to have problems to evaluate some xPath expression.
Not expected - which qualifies for a PMR - or a public work item if non confidential information can be shared
thecorresponding RTC log should here - and we could enable some tracing:
1 - stop the RTC server
2 - rotate the CCM log - rename ccm.log ccm.old.log
3 - edit server/conf/ccm/log4j.properties
4 - add this line at the bottom of the file:
log4j.logger.com.ibm.team.tpt.internal.service.ImpexService=DEBUG
5 - save your changes
6 - start the server
7 - do the import and wait for the issue to show up.
--> collect resulting ccm.log
2 - rotate the CCM log - rename ccm.log ccm.old.log
3 - edit server/conf/ccm/log4j.properties
4 - add this line at the bottom of the file:
log4j.logger.com.ibm.team.tpt.internal.service.ImpexService=DEBUG
5 - save your changes
6 - start the server
7 - do the import and wait for the issue to show up.
--> collect resulting ccm.log
Thanks
Comments
Hi Eric,
Do you think that it could be also useful also to provide MS Project file to try recreating it on site?
Hello Krzys,
well - it depends if the issue is with exporting from RTC to XML / MS Project
or importing from MS Project back into RTC.
For the later, yes, having the MS Project file would help a lot.
Thanks.
Eric.
According to the user it seems they were trying to export from a plan, so there was no Microsoft project file involved. We haven't been able to reproduce the issue with the same plan. Since this is our production environment we'll have to get the change scheduled to do the bounce and update the log4j config. Would it be helpful to go ahead and open a jazz.net defect even though I may not be able to get any more DEBUG logging? This seems like a big deal because what we found is for a period of time we were spiking 3 of our CPUs (1 per hung thread), fortunately we had capacity to handle it, but it could have been worse.
Hello Scott,
unless we have a specific scenario to reproduce the issue in a systematic manner,
I would say it is not useful to open a new defect or a PMR.
Based on the fact we can not investigate / debug a case that we can not reproduce.
On the other hand, you may update the log4 and trace the import / export.
and try and come up with steps to reproduce the issue.
Next time we have a similar issue, we may have more accurate information.
Thanks.
Eric.
Hello Scott and Eric,
Scott - I believe you can try openning the PMR anyway even if you do not have case to reproduce. RTC support is quite smart and might help you in narrowing down the issue and in discovering any patterns.
Do you have maybye any other error messages in the log? Also how often does this error appear you in the log? Do you see any patterns here?
One other answer
Hello Scott,
MSPXMLExportService is the code that enables the export of RTC Plans to MS Project / XML format.
If the thread is stuck, I would think that there is something preventing the operation to complete.
You may want to check with the user(s) performing this kind of operation.
If the problem persists, I would open a PMR - https://jazz.net/help/support/
Hope it helps.
Eric.
Comments
Scott Crouch
Apr 04 '13, 11:22 a.m.This was also tied to high CPU usage for the web server.
We were able to track down from HTTP logs that the user was likely trying to either export or import a plan from the Web UI.
Still not sure what would trigger the problem.