Jazz Forum Welcome to the Jazz Community Forum Connect and collaborate with IBM Engineering experts and users

How to troubleshoot WAS hung threads all pointing to java/utils/zip/Deflate ?

CCM 6.0.6.1 on WAS 9, AIX 7.1 exhibiting extremely poor behavior with high cpu and large number of hung threads.

I took a thread dump and analyzed with IBM Thread Dump And Memory Analyzer and noticed that most of the threads called out in the CPU analysis related directly to hanging threads that are reported in WAS SystemOut.log like:
[2/12/21 16:28:04:990 EST] 00005bf6 ThreadMonitor W   WSVR0605W: Thread "WebContainer : 66 @@ 16:03 jtn <Jakarta Commons-HttpClient/3.1@IPADDRESS> /ccm/service/com.ibm.team.build.internal.common.ITeamBuildService" (000001f2) has been active for 1466696 milliseconds and may be hung.  There is/are 27 thread(s) in total in the server that may be hung

Thread analyzer stack info for WebContainer :66 has a rather lengthy stack trace:

at java/util/zip/Deflater.deflateBytes(Native Method)
at java/util/zip/Deflater.deflate(Deflater.java:455(Compiled Code))
at java/util/zip/Deflater.deflate(Deflater.java:377(Compiled Code))
at java/util/zip/DeflaterOutputStream.deflate(DeflaterOutputStream.java:262(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet$4.deflate(AbstractTeamServerServlet.java:917(Compiled Code))
at java/util/zip/DeflaterOutputStream.write(DeflaterOutputStream.java:222(Compiled Code))
at java/util/zip/GZIPOutputStream.write(GZIPOutputStream.java:156(Compiled Code))
at sun/nio/cs/StreamEncoder.writeBytes(StreamEncoder.java:233(Compiled Code))
at sun/nio/cs/StreamEncoder.implFlushBuffer(StreamEncoder.java:303(Compiled Code))
at sun/nio/cs/StreamEncoder.implFlush(StreamEncoder.java:307(Compiled Code))
at sun/nio/cs/StreamEncoder.flush(StreamEncoder.java:153(Compiled Code))
at java/io/OutputStreamWriter.flush(OutputStreamWriter.java:287(Compiled Code))
at java/io/BufferedWriter.flush(BufferedWriter.java:265(Compiled Code))
at com/ibm/team/repository/common/internal/marshal/impl/MarshallerWriter.flush(MarshallerWriter.java:75(Compiled Code))
at com/ibm/team/repository/common/internal/marshal/impl/EObjectMarshaller.marshalObjectToOutputStream(EObjectMarshaller.java:155(Compiled Code))
at com/ibm/team/repository/common/internal/marshal/impl/WebServicesMarshaller.marshalObjectToOutputStream(WebServicesMarshaller.java:485(Compiled Code))
at com/ibm/team/repository/common/internal/marshal/impl/WebServicesMarshaller.marshalFaultServiceResponse(WebServicesMarshaller.java:374)
at com/ibm/team/repository/servlet/internal/TeamServletResponseHelper.sendFaultResponse(TeamServletResponseHelper.java:66)
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.doPost(AbstractTeamServerServlet.java:955(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:707)
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.handleRequest2(AbstractTeamServerServlet.java:2848(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.handleRequest(AbstractTeamServerServlet.java:2613(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.access$0(AbstractTeamServerServlet.java:2598(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet$1.service(AbstractTeamServerServlet.java:268(Compiled Code))
at com/ibm/team/repository/internal/service/auth/impl/JAuthHandler$1.run(JAuthHandler.java:110(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.handleAuthentication(AbstractTeamServerServlet.java:2285(Compiled Code))
at com/ibm/team/repository/servlet/AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:1928(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:790(Compiled Code))
at org/eclipse/equinox/http/registry/internal/ServletManager$ServletWrapper.service(ServletManager.java:180(Compiled Code))
at org/eclipse/equinox/http/servlet/internal/ServletRegistration.service(ServletRegistration.java:61(Compiled Code))
at org/eclipse/equinox/http/servlet/internal/ProxyServlet.processAlias(ProxyServlet.java:128(Compiled Code))
at org/eclipse/equinox/http/servlet/internal/ProxyServlet.service(ProxyServlet.java:76(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:790(Compiled Code))
at org/eclipse/equinox/servletbridge/BridgeServlet.service(BridgeServlet.java:152(Compiled Code))
at com/ibm/team/repository/server/servletbridge/JazzServlet.service(JazzServlet.java:74(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:790(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1235(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:779(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:478(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:143(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:96(Compiled Code))
at com/ibm/team/repository/server/servletbridge/BridgeFilter.processDelegate(BridgeFilter.java:165(Compiled Code))
at com/ibm/team/repository/server/servletbridge/BridgeFilter.doFilter(BridgeFilter.java:198(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:197(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:90(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:979(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1119(Compiled Code))
at com/ibm/ws/webcontainer/servlet/CacheServletWrapper.handleRequest(CacheServletWrapper.java:82(Compiled Code))
at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:963(Compiled Code))
at com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1817(Compiled Code))
at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:382(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:532(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:318(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpICLReadCallback.complete(HttpICLReadCallback.java:88(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833(Compiled Code))
at com/ibm/ws/tcp/channel/impl/WorkQueueManager.requestComplete(WorkQueueManager.java:558(Compiled Code))
at com/ibm/ws/tcp/channel/impl/WorkQueueManager.attemptIO(WorkQueueManager.java:608(Compiled Code))
at com/ibm/ws/tcp/channel/impl/WorkQueueManager.workerRun(WorkQueueManager.java:985(Compiled Code))
at com/ibm/ws/tcp/channel/impl/WorkQueueManager$Worker.run(WorkQueueManager.java:1074(Compiled Code))
at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1909(Compiled Code))


Anyone have any suggestions ?

0 votes

Comments

I'd consider contacting support, Kevin. 


Accepted answer

Permanent link
I have done that and I was directed to this work around

This indeed has helped as the worst offender hasn't added new never finishing threads in more than a week.
Ralph Schoon selected this answer as the correct answer

0 votes

Your answer

Register or log in to post your answer.

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

Search context
Follow this question

By Email: 

Once you sign in you will be able to subscribe for any updates here.

By RSS:

Answers
Answers and Comments
Question details
× 7,497
× 1,327

Question asked: Feb 15 '21, 1:52 p.m.

Question was seen: 1,189 times

Last updated: Feb 24 '21, 1:42 a.m.

Confirmation Cancel Confirm