It's all about the answers!

Ask a question

Failed Build stays in 'pending' state if there is a connection timeout


Lukas Steiger (3131525) | asked Mar 27 '14, 3:58 a.m.
 Hi there, 

Yesterday we had to completely restart the CCM application on our jazz server. Since then, the application (WebUI and Eclipse Client) show one build as 'pending'. Looking into the logs, I have seen that there was an exception (which is attached below) and therefore, the execution of this build was stopped. Other builds were successfully executed on this build engine afterwards, so luckily it did not block the build enigne. However, there must be something wrong that the build is still in the 'pending' state after 21 hours.

This ANT build is configured to execute every 15 minutes, but only if there are changes accepted. The was no change in source code for a couple of weeks now. The build engine is JBE 4.0.5, the server is 4.0.5 as well.

Have you any idea what the possible issue of this behavior could be? Shall I raise a defect against it?

Thanks in advance
Lukas



The exception thrown by JBE is the following:
2014-03-26 11:28:17 [Jazz build engine] 
2014-03-26 11:28:17 [Jazz build engine] Substituted the following build property variables:
2014-03-26 11:28:17 [Jazz build engine] team.scm.fetchDestination = ${SERVER_BASE_PATH}\projects\jazz-extensions\timebox-build   -->   team.scm.fetchDestination = D:\CLM-Build-Jobs\projects\jazz-extensions\timebox-build
2014-03-26 11:28:17 [Jazz build engine] 
2014-03-26 11:28:17 [Jazz build engine] 
2014-03-26 11:28:17 [Jazz build engine] Substituted the following configuration element property variables:
2014-03-26 11:28:17 [Jazz build engine] com.ibm.team.build.ant : com.ibm.team.build.ant.buildFile = ${SERVER_BASE_PATH}\projects\jazz-extensions\timebox-build\com.siemens.bt.jazz.timebox.viewlet.web\buildengine.xml   -->   com.ibm.team.build.ant.buildFile = D:\CLM-Build-Jobs\projects\jazz-extensions\timebox-build\com.siemens.bt.jazz.timebox.viewlet.web\buildengine.xml
2014-03-26 11:28:17 [Jazz build engine] com.ibm.team.build.ant : com.ibm.team.build.ant.javaHome = ${SERVER_BASE_PATH}\tools\java\jdk1.7   -->   com.ibm.team.build.ant.javaHome = D:\CLM-Build-Jobs\tools\java\jdk1.7
2014-03-26 11:28:17 [Jazz build engine] com.ibm.team.build.ant : com.ibm.team.build.ant.workingDir = ${SERVER_BASE_PATH}\projects\jazz-extensions\timebox-build\com.siemens.bt.jazz.timebox.viewlet.web   -->   com.ibm.team.build.ant.workingDir = D:\CLM-Build-Jobs\projects\jazz-extensions\timebox-build\com.siemens.bt.jazz.timebox.viewlet.web
2014-03-26 11:28:17 [Jazz build engine] 
2014-03-26 11:28:17 [Jazz build engine] running on host: ************
2014-03-26 11:28:17 [Jazz build engine] Should build occur?
2014-03-26 11:28:17 [Jazz build engine] Checking for changes incoming to workspace "Timebox Build Workspace" ...
com.ibm.team.repository.common.LicenseNotGrantedException: Unable to obtain license to read SCM related data: Timeout waiting for connection.
at com.ibm.team.scm.service.internal.AbstractScmService.assertLicense(AbstractScmService.java:276)
at com.ibm.team.scm.service.internal.ScmService.refreshWorkspaces(ScmService.java:4475)
at sun.reflect.GeneratedMethodAccessor137.invoke(null)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at com.sun.proxy.$Proxy708.refreshWorkspaces(null)
at sun.reflect.GeneratedMethodAccessor137.invoke(null)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleMethod(AbstractTeamServerServlet.java:1421)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.executeMethod(AbstractTeamServerServlet.java:956)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.doPost(AbstractTeamServerServlet.java:729)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest2(AbstractTeamServerServlet.java:2356)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest(AbstractTeamServerServlet.java:2153)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.access$0(AbstractTeamServerServlet.java:2138)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet$1.service(AbstractTeamServerServlet.java:219)
at com.ibm.team.repository.internal.service.auth.impl.JAuthHandler$1.run(JAuthHandler.java:109)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:1792)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:180)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:126)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:76)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at org.eclipse.equinox.servletbridge.BridgeServlet.service(BridgeServlet.java:120)
at com.ibm.team.repository.server.servletbridge.JazzServlet.service(JazzServlet.java:74)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at com.ibm.team.repository.server.servletbridge.BridgeFilter.processDelegate(BridgeFilter.java:165)
at com.ibm.team.repository.server.servletbridge.BridgeFilter.doFilter(BridgeFilter.java:198)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:960)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1064)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:909)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:84)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1700)
at com.ibm.team.repository.common.internal.marshal.util.MarshallerUtil.decodeExceptions(MarshallerUtil.java:340)
at com.ibm.team.repository.common.internal.marshal.util.MarshallerUtil.decodeExceptions(MarshallerUtil.java:310)
at com.ibm.team.repository.common.internal.marshal.util.MarshallerUtil.decodeFault(MarshallerUtil.java:275)
at com.ibm.team.repository.transport.client.RemoteTeamService.constructExceptionFromFault(RemoteTeamService.java:609)
at com.ibm.team.repository.transport.client.RemoteTeamService.executeMethod(RemoteTeamService.java:493)
at com.ibm.team.repository.transport.client.RemoteTeamService.invoke(RemoteTeamService.java:194)
at com.ibm.team.repository.transport.client.ServiceInvocationHandler.invoke(ServiceInvocationHandler.java:43)
at com.sun.proxy.$Proxy22.refreshWorkspaces(Unknown Source)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.ibm.team.repository.client.internal.ServiceInterfaceProxy.invokeServiceCall(ServiceInterfaceProxy.java:254)
at com.ibm.team.repository.client.internal.ServiceInterfaceProxy.invoke(ServiceInterfaceProxy.java:110)
at com.sun.proxy.$Proxy22.refreshWorkspaces(Unknown Source)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.ibm.team.scm.client.internal.ScmServiceInterfaceProxy$2.run(ScmServiceInterfaceProxy.java:168)
at com.ibm.team.repository.client.internal.TeamRepository$3.run(TeamRepository.java:1320)
at com.ibm.team.repository.common.transport.CancelableCaller.call(CancelableCaller.java:79)
at com.ibm.team.repository.client.internal.TeamRepository.callCancelableService(TeamRepository.java:1315)
at com.ibm.team.scm.client.internal.ScmClientLibraryContext.callCancelableService(ScmClientLibraryContext.java:71)
at com.ibm.team.scm.client.internal.ScmServiceInterfaceProxy.invokeCancelableService(ScmServiceInterfaceProxy.java:164)
at com.ibm.team.scm.client.internal.ScmServiceInterfaceProxy.invoke(ScmServiceInterfaceProxy.java:92)
at com.sun.proxy.$Proxy22.refreshWorkspaces(Unknown Source)
at com.ibm.team.scm.client.internal.WorkspaceManager.getWorkspaceConnection(WorkspaceManager.java:868)
at com.ibm.team.build.internal.scm.BuildWorkspaceDescriptor.getConnection(BuildWorkspaceDescriptor.java:165)
at com.ibm.team.build.internal.scm.SourceControlUtility.checkForIncoming(SourceControlUtility.java:912)
at com.ibm.team.build.internal.engine.JazzScmPreBuildParticipant.shouldBuild(JazzScmPreBuildParticipant.java:87)
at com.ibm.team.build.internal.engine.BuildLoop.shouldBuild(BuildLoop.java:1294)
at com.ibm.team.build.internal.engine.BuildLoop.runLoop(BuildLoop.java:400)
at com.ibm.team.build.internal.engine.BuildLoop$1.run(BuildLoop.java:212)
at java.lang.Thread.run(Thread.java:744)
Caused by: com.ibm.team.repository.common.LicenseNotGrantedException: Timeout waiting for connection
at com.ibm.team.repository.service.compatibility.license.internal.LicenseService.assertLicenseRemotely(LicenseService.java:168)
at com.ibm.team.repository.service.compatibility.license.internal.LicenseService.assertLicense(LicenseService.java:144)
at com.ibm.team.repository.service.compatibility.license.internal.LicenseService.assertLicenseForContributor(LicenseService.java:174)
at com.ibm.team.repository.service.compatibility.license.internal.LicenseService.assertLicense(LicenseService.java:454)
at sun.reflect.GeneratedMethodAccessor141.invoke(null)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at com.sun.proxy.$Proxy253.assertLicense(null)
at com.ibm.team.repository.service.internal.license.jts.JtsLicenseService.assertLicense(JtsLicenseService.java:38)
at sun.reflect.GeneratedMethodAccessor140.invoke(null)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at com.sun.proxy.$Proxy404.assertLicense(null)
at com.ibm.team.scm.service.internal.AbstractScmService.assertLicense(AbstractScmService.java:266)
... 92 more
Caused by: com.ibm.team.repository.common.TeamRepositoryException: Timeout waiting for connection
at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClient.doCommunicate(CrossServerHttpClient.java:196)
at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClient.send(CrossServerHttpClient.java:447)
at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClient.send(CrossServerHttpClient.java:336)
at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClient.send(CrossServerHttpClient.java:460)
at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClientService.send(CrossServerHttpClientService.java:68)
at sun.reflect.GeneratedMethodAccessor123.invoke(null)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at com.sun.proxy.$Proxy252.send(null)
at com.ibm.team.repository.service.compatibility.AbstractCompatibilityService.send(AbstractCompatibilityService.java:56)
at com.ibm.team.repository.service.compatibility.license.internal.LicenseService.assertLicenseRemotely(LicenseService.java:165)
... 111 more
Caused by: java.io.InterruptedIOException: <Remote exception was of type 'org.apache.http.conn.ConnectionPoolTimeoutException'>
Timeout waiting for connection
at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:417)
at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300)
at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:401)
at com.ibm.team.jfs.app.http.client.internal.ExtensibleRequestDirector.execute(ExtensibleRequestDirector.java:149)
at com.ibm.team.jfs.app.http.client.cache.HttpClientCache.executeCacheMiss(HttpClientCache.java:217)
at com.ibm.team.jfs.app.http.client.cache.HttpClientCache.executeCacheMiss(HttpClientCache.java:201)
at com.ibm.team.jfs.app.http.client.cache.HttpClientCache.execute(HttpClientCache.java:140)
at com.ibm.team.jfs.app.http.client.cache.HttpClientCache.execute(HttpClientCache.java:118)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:776)
at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClient.doCommunicate(CrossServerHttpClient.java:185)
... 124 more


Comments
Scott Cowan commented Mar 27 '14, 9:18 a.m.
JAZZ DEVELOPER

@heatherf I wonder if this exception is a coincidence, or if there's a new case we need to add to our improved retry logic in 4.0.5?

Make JBE's SCM participant resilient to temporary server outages and retryable request failures (265141)

Accepted answer


permanent link
Nick Edgar (6.5k711) | answered Mar 27 '14, 1:07 p.m.
JAZZ DEVELOPER
Lukas Steiger selected this answer as the correct answer

One other answer



permanent link
Abraham Sweiss (2.4k1131) | answered Mar 27 '14, 12:47 p.m.
Based on the error:
com.ibm.team.repository.common.LicenseNotGrantedException: Unable to obtain license to read SCM related data: Timeout waiting for connection.
This seems to indicate a problem accessing jts to validate the user running the builds has the correct authority
I would review the jts logs to see if there were any issues at the time this exception was thrown.

Now if you feel this scenario should be handled more gracefully in RTC, yes open a defect workitem


Your answer


Register or to post your answer.