It's all about the answers!

Ask a question
QuestionsTagsUsersBadges
Follow this question
Question details

×7,382
×5,996
×1,210

question asked: Feb 15 '21, 1:52 p.m.
question was seen: 892 times
last updated: Feb 24 '21, 1:42 a.m.

Related questions

FAQ - How this Forum works

How to troubleshoot WAS hung threads all pointing to java/utils/zip/Deflate ?

0
Kevin Ramer (4.5k●8●183●200) | asked Feb 15 '21, 1:52 p.m.
edited Feb 15 '21, 1:53 p.m.
CCM 6.0.6.1 on WAS 9, AIX 7.1 exhibiting extremely poor behavior with high cpu and large number of hung threads.

I took a thread dump and analyzed with IBM Thread Dump And Memory Analyzer and noticed that most of the threads called out in the CPU analysis related directly to hanging threads that are reported in WAS SystemOut.log like:
[2/12/21 16:28:04:990 EST] 00005bf6 ThreadMonitor W   WSVR0605W: Thread "WebContainer : 66 @@ 16:03 jtn <Jakarta Commons-HttpClient/3.1@IPADDRESS> /ccm/service/com.ibm.team.build.internal.common.ITeamBuildService" (000001f2) has been active for 1466696 milliseconds and may be hung.  There is/are 27 thread(s) in total in the server that may be hung

Thread analyzer stack info for WebContainer :66 has a rather lengthy stack trace:

at java/util/zip/Deflater.deflateBytes(Native Method)
at java/util/zip/Deflater.deflate(Deflater.java:455(Compiled Code))
at java/util/zip/Deflater.deflate(Deflater.java:377(Compiled Code))
at java/util/zip/DeflaterOutputStream.deflate(DeflaterOutputStream.java:262(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet$4.deflate(AbstractTeamServerServlet.java:917(Compiled Code))
at java/util/zip/DeflaterOutputStream.write(DeflaterOutputStream.java:222(Compiled Code))
at java/util/zip/GZIPOutputStream.write(GZIPOutputStream.java:156(Compiled Code))
at sun/nio/cs/StreamEncoder.writeBytes(StreamEncoder.java:233(Compiled Code))
at sun/nio/cs/StreamEncoder.implFlushBuffer(StreamEncoder.java:303(Compiled Code))
at sun/nio/cs/StreamEncoder.implFlush(StreamEncoder.java:307(Compiled Code))
at sun/nio/cs/StreamEncoder.flush(StreamEncoder.java:153(Compiled Code))
at java/io/OutputStreamWriter.flush(OutputStreamWriter.java:287(Compiled Code))
at java/io/BufferedWriter.flush(BufferedWriter.java:265(Compiled Code))
at com/ibm/team/repository/common/internal/marshal/impl/MarshallerWriter.flush(MarshallerWriter.java:75(Compiled Code))
at com/ibm/team/repository/common/internal/marshal/impl/EObjectMarshaller.marshalObjectToOutputStream(EObjectMarshaller.java:155(Compiled Code))
at com/ibm/team/repository/common/internal/marshal/impl/WebServicesMarshaller.marshalObjectToOutputStream(WebServicesMarshaller.java:485(Compiled Code))
at com/ibm/team/repository/common/internal/marshal/impl/WebServicesMarshaller.marshalFaultServiceResponse(WebServicesMarshaller.java:374)
at com/ibm/team/repository/servlet/internal/TeamServletResponseHelper.sendFaultResponse(TeamServletResponseHelper.java:66)
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.doPost(AbstractTeamServerServlet.java:955(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:707)
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.handleRequest2(AbstractTeamServerServlet.java:2848(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.handleRequest(AbstractTeamServerServlet.java:2613(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.access$0(AbstractTeamServerServlet.java:2598(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet$1.service(AbstractTeamServerServlet.java:268(Compiled Code))
at com/ibm/team/repository/internal/service/auth/impl/JAuthHandler$1.run(JAuthHandler.java:110(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.handleAuthentication(AbstractTeamServerServlet.java:2285(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:1928(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:790(Compiled Code))
at org/eclipse/equinox/http/registry/internal/ServletManager$ServletWrapper.service(ServletManager.java:180(Compiled Code))
at org/eclipse/equinox/http/servlet/internal/ServletRegistration.service(ServletRegistration.java:61(Compiled Code))
at org/eclipse/equinox/http/servlet/internal/ProxyServlet.processAlias(ProxyServlet.java:128(Compiled Code))
at org/eclipse/equinox/http/servlet/internal/ProxyServlet.service(ProxyServlet.java:76(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:790(Compiled Code))
at org/eclipse/equinox/servletbridge/BridgeServlet.service(BridgeServlet.java:152(Compiled Code))
at com/ibm/team/repository/server/servletbridge/JazzServlet.service(JazzServlet.java:74(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:790(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1235(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:779(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:478(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:143(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:96(Compiled Code))
at com/ibm/team/repository/server/servletbridge/BridgeFilter.processDelegate(BridgeFilter.java:165(Compiled Code))
at com/ibm/team/repository/server/servletbridge/BridgeFilter.doFilter(BridgeFilter.java:198(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:197(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:90(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:979(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1119(Compiled Code))
at com/ibm/ws/webcontainer/servlet/CacheServletWrapper.handleRequest(CacheServletWrapper.java:82(Compiled Code))
at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:963(Compiled Code))
at com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1817(Compiled Code))
at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:382(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:532(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:318(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpICLReadCallback.complete(HttpICLReadCallback.java:88(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833(Compiled Code))
at com/ibm/ws/tcp/channel/impl/WorkQueueManager.requestComplete(WorkQueueManager.java:558(Compiled Code))
at com/ibm/ws/tcp/channel/impl/WorkQueueManager.attemptIO(WorkQueueManager.java:608(Compiled Code))
at com/ibm/ws/tcp/channel/impl/WorkQueueManager.workerRun(WorkQueueManager.java:985(Compiled Code))
at com/ibm/ws/tcp/channel/impl/WorkQueueManager$Worker.run(WorkQueueManager.java:1074(Compiled Code))
at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1909(Compiled Code))


Anyone have any suggestions ?

Comments
Ralph Schoon commented Feb 16 '21, 3:00 a.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

I'd consider contacting support, Kevin. 

Accepted answer

0
permanent link
Kevin Ramer (4.5k●8●183●200) | answered Feb 23 '21, 3:08 p.m.
I have done that and I was directed to this work around

This indeed has helped as the worst offender hasn't added new never finishing threads in more than a week.
Ralph Schoon selected this answer as the correct answer

Your answer

Register or to post your answer.


Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.