How to troubleshoot WAS hung threads all pointing to java/utils/zip/Deflate ?
CCM 6.0.6.1 on WAS 9, AIX 7.1 exhibiting extremely poor behavior with high cpu and large number of hung threads.
I took a thread dump and analyzed with IBM Thread Dump And Memory Analyzer and noticed that most of the threads called out in the CPU analysis related directly to hanging threads that are reported in WAS SystemOut.log like:
[2/12/21 16:28:04:990 EST] 00005bf6 ThreadMonitor W WSVR0605W: Thread "WebContainer : 66 @@ 16:03 jtn <Jakarta Commons-HttpClient/3.1@IPADDRESS> /ccm/service/com.ibm.team.build.internal.common.ITeamBuildService" (000001f2) has been active for 1466696 milliseconds and may be hung. There is/are 27 thread(s) in total in the server that may be hung
Thread analyzer stack info for WebContainer :66 has a rather lengthy stack trace:
at java/util/zip/Deflater.deflateBytes(Native Method)
at java/util/zip/Deflater.deflate(Deflater.java:455(Compiled Code))
at java/util/zip/Deflater.deflate(Deflater.java:377(Compiled Code))
at java/util/zip/DeflaterOutputStream.deflate(DeflaterOutputStream.java:262(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet$4.deflate(AbstractTeamServerServlet.java:917(Compiled Code))
at java/util/zip/DeflaterOutputStream.write(DeflaterOutputStream.java:222(Compiled Code))
at java/util/zip/GZIPOutputStream.write(GZIPOutputStream.java:156(Compiled Code))
at sun/nio/cs/StreamEncoder.writeBytes(StreamEncoder.java:233(Compiled Code))
at sun/nio/cs/StreamEncoder.implFlushBuffer(StreamEncoder.java:303(Compiled Code))
at sun/nio/cs/StreamEncoder.implFlush(StreamEncoder.java:307(Compiled Code))
at sun/nio/cs/StreamEncoder.flush(StreamEncoder.java:153(Compiled Code))
at java/io/OutputStreamWriter.flush(OutputStreamWriter.java:287(Compiled Code))
at java/io/BufferedWriter.flush(BufferedWriter.java:265(Compiled Code))
at com/ibm/team/repository/common/internal/marshal/impl/MarshallerWriter.flush(MarshallerWriter.java:75(Compiled Code))
at com/ibm/team/repository/common/internal/marshal/impl/EObjectMarshaller.marshalObjectToOutputStream(EObjectMarshaller.java:155(Compiled Code))
at com/ibm/team/repository/common/internal/marshal/impl/WebServicesMarshaller.marshalObjectToOutputStream(WebServicesMarshaller.java:485(Compiled Code))
at com/ibm/team/repository/common/internal/marshal/impl/WebServicesMarshaller.marshalFaultServiceResponse(WebServicesMarshaller.java:374)
at com/ibm/team/repository/servlet/internal/TeamServletResponseHelper.sendFaultResponse(TeamServletResponseHelper.java:66)
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.doPost(AbstractTeamServerServlet.java:955(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:707)
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.handleRequest2(AbstractTeamServerServlet.java:2848(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.handleRequest(AbstractTeamServerServlet.java:2613(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.access$0(AbstractTeamServerServlet.java:2598(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet$1.service(AbstractTeamServerServlet.java:268(Compiled Code))
at com/ibm/team/repository/internal/service/auth/impl/JAuthHandler$1.run(JAuthHandler.java:110(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.handleAuthentication(AbstractTeamServerServlet.java:2285(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:1928(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:790(Compiled Code))
at org/eclipse/equinox/http/registry/internal/ServletManager$ServletWrapper.service(ServletManager.java:180(Compiled Code))
at org/eclipse/equinox/http/servlet/internal/ServletRegistration.service(ServletRegistration.java:61(Compiled Code))
at org/eclipse/equinox/http/servlet/internal/ProxyServlet.processAlias(ProxyServlet.java:128(Compiled Code))
at org/eclipse/equinox/http/servlet/internal/ProxyServlet.service(ProxyServlet.java:76(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:790(Compiled Code))
at org/eclipse/equinox/servletbridge/BridgeServlet.service(BridgeServlet.java:152(Compiled Code))
at com/ibm/team/repository/server/servletbridge/JazzServlet.service(JazzServlet.java:74(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:790(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1235(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:779(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:478(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:143(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:96(Compiled Code))
at com/ibm/team/repository/server/servletbridge/BridgeFilter.processDelegate(BridgeFilter.java:165(Compiled Code))
at com/ibm/team/repository/server/servletbridge/BridgeFilter.doFilter(BridgeFilter.java:198(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:197(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:90(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:979(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1119(Compiled Code))
at com/ibm/ws/webcontainer/servlet/CacheServletWrapper.handleRequest(CacheServletWrapper.java:82(Compiled Code))
at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:963(Compiled Code))
at com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1817(Compiled Code))
at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:382(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:532(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:318(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpICLReadCallback.complete(HttpICLReadCallback.java:88(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833(Compiled Code))
at com/ibm/ws/tcp/channel/impl/WorkQueueManager.requestComplete(WorkQueueManager.java:558(Compiled Code))
at com/ibm/ws/tcp/channel/impl/WorkQueueManager.attemptIO(WorkQueueManager.java:608(Compiled Code))
at com/ibm/ws/tcp/channel/impl/WorkQueueManager.workerRun(WorkQueueManager.java:985(Compiled Code))
at com/ibm/ws/tcp/channel/impl/WorkQueueManager$Worker.run(WorkQueueManager.java:1074(Compiled Code))
at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1909(Compiled Code))
Anyone have any suggestions ?
Accepted answer
I have done that and I was directed to this work around
This indeed has helped as the worst offender hasn't added new never finishing threads in more than a week.
Comments
Ralph Schoon
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Feb 16 '21, 3:00 a.m.I'd consider contacting support, Kevin.