Hung thread on com.ibm.team.tpt.internal.service.ImpexService.service?

Scott Crouch (485●3●23●26) | asked Apr 04 '13, 10:57 a.m.

We're occasionally seeing hung thread alerts from WebSphere, when we look in the active services of CCM we see active services with long running times for com.ibm.team.tpt.internal.service.ImpexService.service.

Has anybody seen this before? Any ideas what could be triggering it?

The hung thread stack trace appears to be:

[4/4/13 9:40:23:589 CDT] 00000046 ThreadMonitor W WSVR0605W: Thread "WebContainer : 8922" (0000be8e) has been active for 1495600 milliseconds and may be hung. There is/are 2 thread(s) in total in the server that may be hung.

at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.addNode(Unknown Source)

at org.apache.xml.dtm.ref.dom2dtm.DOM2DTM.nextNode(Unknown Source)

at org.apache.xml.dtm.ref.DTMDefaultBase._nextsib(Unknown Source)

at org.apache.xml.dtm.ref.DTMDefaultBase.getNextSibling(Unknown Source)

at org.apache.xml.dtm.ref.DTMDefaultBaseTraversers$ChildTraverser.next(Unknown Source)

at org.apache.xpath.axes.AxesWalker.getNextNode(Unknown Source)

at org.apache.xpath.axes.AxesWalker.nextNode(Unknown Source)

at org.apache.xpath.axes.WalkingIterator.nextNode(Unknown Source)

at org.apache.xpath.axes.NodeSequence.nextNode(Unknown Source)

at org.apache.xpath.axes.NodeSequence.runTo(Unknown Source)

at org.apache.xml.dtm.ref.DTMNodeList.<init>(Unknown Source)

at org.apache.xpath.objects.XNodeSet.nodelist(Unknown Source)

at org.apache.xpath.jaxp.XPathExpressionImpl.xObjectToObject(Unknown Source)

at org.apache.xpath.jaxp.XPathExpressionImpl.evaluate(Unknown Source)

at com.ibm.team.tpt.internal.service.util.ImpexUtil.retrieveDataList(ImpexUtil.java:608)

at com.ibm.team.tpt.internal.service.impex.AbstractXMLExportService.generateTaskInfo(AbstractXMLExportService.java:200)

at com.ibm.team.tpt.internal.service.impex.AbstractXMLExportService.generateTaskInfo(AbstractXMLExportService.java:312)

at com.ibm.team.tpt.internal.service.impex.AbstractXMLExportService.generateTasksInfo(AbstractXMLExportService.java:422)

at com.ibm.team.tpt.internal.service.impex.AbstractXMLExportService.generateDocumentFromObjects(AbstractXMLExportService.java:88)

at com.ibm.team.tpt.internal.service.impex.MSPXMLExportService.doExport(MSPXMLExportService.java:38)

at com.ibm.team.tpt.internal.service.ImpexService.perform_GET(ImpexService.java:89)

at com.ibm.team.repository.service.TeamRawService.service(TeamRawService.java:82)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

at java.lang.reflect.Method.invoke(Method.java:611)

at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)

at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)

at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)

at $Proxy1116.service(Unknown Source)

at com.ibm.team.repository.servlet.AbstractTeamServerServlet.doRestService(AbstractTeamServerServlet.java:914)

at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest2(AbstractTeamServerServlet.java:2100)

at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest(AbstractTeamServerServlet.java:1924)

at com.ibm.team.repository.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:1800)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)

WAS 7, RHEL, RTC 4.0.1.

Comments

Scott Crouch commented Apr 04 '13, 11:22 a.m.

This was also tied to high CPU usage for the web server.

We were able to track down from HTTP logs that the user was likely trying to either export or import a plan from the Web UI.

Still not sure what would trigger the problem.

Can open a defect/PMR if that's the only answer, just wondering if anybody had seen this before or knew of an existing defect.

Accepted answer

permanent link

Eric Jodet (6.3k●5●111●120) | answered Apr 04 '13, 11:37 a.m.
JAZZ DEVELOPER

looking at the stack:

at org.apache.xpath.jaxp.XPathExpressionImpl.evaluate(Unknown Source)

at com.ibm.team.tpt.internal.service.util.ImpexUtil.retrieveDataList(ImpexUtil.java:608)

we seem to have problems to evaluate some xPath expression.

Not expected - which qualifies for a PMR - or a public work item if non confidential information can be shared

thecorresponding RTC log should here - and we could enable some tracing:

1 - stop the RTC server
2 - rotate the CCM log - rename ccm.log ccm.old.log
3 - edit server/conf/ccm/log4j.properties
4 - add this line at the bottom of the file:
log4j.logger.com.ibm.team.tpt.internal.service.ImpexService=DEBUG
5 - save your changes
6 - start the server
7 - do the import and wait for the issue to show up.
--> collect resulting ccm.log

Thanks

Scott Crouch selected this answer as the correct answer

Comments

Krzysztof Kaźmierczyk commented Apr 05 '13, 2:37 a.m.

Hi Eric,
Do you think that it could be also useful also to provide MS Project file to try recreating it on site?

Eric Jodet commented Apr 05 '13, 3:36 a.m.

JAZZ DEVELOPER

Hello Krzys,

well - it depends if the issue is with exporting from RTC to XML / MS Project

or importing from MS Project back into RTC.

For the later, yes, having the MS Project file would help a lot.

Thanks.

Eric.

Scott Crouch commented Apr 05 '13, 8:43 a.m.

According to the user it seems they were trying to export from a plan, so there was no Microsoft project file involved. We haven't been able to reproduce the issue with the same plan. Since this is our production environment we'll have to get the change scheduled to do the bounce and update the log4j config. Would it be helpful to go ahead and open a jazz.net defect even though I may not be able to get any more DEBUG logging? This seems like a big deal because what we found is for a period of time we were spiking 3 of our CPUs (1 per hung thread), fortunately we had capacity to handle it, but it could have been worse.

Eric Jodet commented Apr 05 '13, 9:05 a.m.

JAZZ DEVELOPER

Hello Scott,

unless we have a specific scenario to reproduce the issue in a systematic manner,

I would say it is not useful to open a new defect or a PMR.

Based on the fact we can not investigate / debug a case that we can not reproduce.

On the other hand, you may update the log4 and trace the import / export.

and try and come up with steps to reproduce the issue.

Next time we have a similar issue, we may have more accurate information.

Thanks.

Eric.

Krzysztof Kaźmierczyk commented Apr 08 '13, 3:48 a.m.

Hello Scott and Eric,
Scott - I believe you can try openning the PMR anyway even if you do not have case to reproduce. RTC support is quite smart and might help you in narrowing down the issue and in discovering any patterns.

Do you have maybye any other error messages in the log? Also how often does this error appear you in the log? Do you see any patterns here?

One other answer

Most liked answers|Newest answers ↑|Oldest answers

permanent link

Eric Jodet (6.3k●5●111●120) | answered Apr 04 '13, 11:22 a.m.
JAZZ DEVELOPER

Hello Scott,

MSPXMLExportService is the code that enables the export of RTC Plans to MS Project / XML format.

If the thread is stuck, I would think that there is something preventing the operation to complete.

You may want to check with the user(s) performing this kind of operation.

If the problem persists, I would open a PMR - https://jazz.net/help/support/

Hope it helps.

Eric.

It's all about the answers!

Related questions

Hung thread on com.ibm.team.tpt.internal.service.ImpexService.service?

Accepted answer

One other answer

Your answer