Failed Build stays in 'pending' state if there is a connection timeout
Hi there,
Yesterday we had to completely restart the CCM application on our jazz server. Since then, the application (WebUI and Eclipse Client) show one build as 'pending'. Looking into the logs, I have seen that there was an exception (which is attached below) and therefore, the execution of this build was stopped. Other builds were successfully executed on this build engine afterwards, so luckily it did not block the build enigne. However, there must be something wrong that the build is still in the 'pending' state after 21 hours.
This ANT build is configured to execute every 15 minutes, but only if there are changes accepted. The was no change in source code for a couple of weeks now. The build engine is JBE 4.0.5, the server is 4.0.5 as well.
Have you any idea what the possible issue of this behavior could be? Shall I raise a defect against it?
Thanks in advance
Lukas
The exception thrown by JBE is the following:
2014-03-26 11:28:17 [Jazz build engine]2014-03-26 11:28:17 [Jazz build engine] Substituted the following build property variables:2014-03-26 11:28:17 [Jazz build engine] team.scm.fetchDestination = ${SERVER_BASE_PATH}\projects\jazz-extensions\timebox-build --> team.scm.fetchDestination = D:\CLM-Build-Jobs\projects\jazz-extensions\timebox-build2014-03-26 11:28:17 [Jazz build engine]2014-03-26 11:28:17 [Jazz build engine]2014-03-26 11:28:17 [Jazz build engine] Substituted the following configuration element property variables:2014-03-26 11:28:17 [Jazz build engine] com.ibm.team.build.ant : com.ibm.team.build.ant.buildFile = ${SERVER_BASE_PATH}\projects\jazz-extensions\timebox-build\com.siemens.bt.jazz.timebox.viewlet.web\buildengine.xml --> com.ibm.team.build.ant.buildFile = D:\CLM-Build-Jobs\projects\jazz-extensions\timebox-build\com.siemens.bt.jazz.timebox.viewlet.web\buildengine.xml2014-03-26 11:28:17 [Jazz build engine] com.ibm.team.build.ant : com.ibm.team.build.ant.javaHome = ${SERVER_BASE_PATH}\tools\java\jdk1.7 --> com.ibm.team.build.ant.javaHome = D:\CLM-Build-Jobs\tools\java\jdk1.72014-03-26 11:28:17 [Jazz build engine] com.ibm.team.build.ant : com.ibm.team.build.ant.workingDir = ${SERVER_BASE_PATH}\projects\jazz-extensions\timebox-build\com.siemens.bt.jazz.timebox.viewlet.web --> com.ibm.team.build.ant.workingDir = D:\CLM-Build-Jobs\projects\jazz-extensions\timebox-build\com.siemens.bt.jazz.timebox.viewlet.web2014-03-26 11:28:17 [Jazz build engine]2014-03-26 11:28:17 [Jazz build engine] running on host: ************2014-03-26 11:28:17 [Jazz build engine] Should build occur?2014-03-26 11:28:17 [Jazz build engine] Checking for changes incoming to workspace "Timebox Build Workspace" ...com.ibm.team.repository.common.LicenseNotGrantedException: Unable to obtain license to read SCM related data: Timeout waiting for connection.at com.ibm.team.scm.service.internal.AbstractScmService.assertLicense(AbstractScmService.java:276)at com.ibm.team.scm.service.internal.ScmService.refreshWorkspaces(ScmService.java:4475)at sun.reflect.GeneratedMethodAccessor137.invoke(null)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)at java.lang.reflect.Method.invoke(Method.java:611)at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)at com.sun.proxy.$Proxy708.refreshWorkspaces(null)at sun.reflect.GeneratedMethodAccessor137.invoke(null)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)at java.lang.reflect.Method.invoke(Method.java:611)at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleMethod(AbstractTeamServerServlet.java:1421)at com.ibm.team.repository.servlet.AbstractTeamServerServlet.executeMethod(AbstractTeamServerServlet.java:956)at com.ibm.team.repository.servlet.AbstractTeamServerServlet.doPost(AbstractTeamServerServlet.java:729)at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest2(AbstractTeamServerServlet.java:2356)at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest(AbstractTeamServerServlet.java:2153)at com.ibm.team.repository.servlet.AbstractTeamServerServlet.access$0(AbstractTeamServerServlet.java:2138)at com.ibm.team.repository.servlet.AbstractTeamServerServlet$1.service(AbstractTeamServerServlet.java:219)at com.ibm.team.repository.internal.service.auth.impl.JAuthHandler$1.run(JAuthHandler.java:109)at com.ibm.team.repository.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:1792)at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)at org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:180)at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:126)at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:76)at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)at org.eclipse.equinox.servletbridge.BridgeServlet.service(BridgeServlet.java:120)at com.ibm.team.repository.server.servletbridge.JazzServlet.service(JazzServlet.java:74)at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)at com.ibm.team.repository.server.servletbridge.BridgeFilter.processDelegate(BridgeFilter.java:165)at com.ibm.team.repository.server.servletbridge.BridgeFilter.doFilter(BridgeFilter.java:198)at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:960)at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1064)at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:909)at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:84)at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1700)at com.ibm.team.repository.common.internal.marshal.util.MarshallerUtil.decodeExceptions(MarshallerUtil.java:340)at com.ibm.team.repository.common.internal.marshal.util.MarshallerUtil.decodeExceptions(MarshallerUtil.java:310)at com.ibm.team.repository.common.internal.marshal.util.MarshallerUtil.decodeFault(MarshallerUtil.java:275)at com.ibm.team.repository.transport.client.RemoteTeamService.constructExceptionFromFault(RemoteTeamService.java:609)at com.ibm.team.repository.transport.client.RemoteTeamService.executeMethod(RemoteTeamService.java:493)at com.ibm.team.repository.transport.client.RemoteTeamService.invoke(RemoteTeamService.java:194)at com.ibm.team.repository.transport.client.ServiceInvocationHandler.invoke(ServiceInvocationHandler.java:43)at com.sun.proxy.$Proxy22.refreshWorkspaces(Unknown Source)at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at com.ibm.team.repository.client.internal.ServiceInterfaceProxy.invokeServiceCall(ServiceInterfaceProxy.java:254)at com.ibm.team.repository.client.internal.ServiceInterfaceProxy.invoke(ServiceInterfaceProxy.java:110)at com.sun.proxy.$Proxy22.refreshWorkspaces(Unknown Source)at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at com.ibm.team.scm.client.internal.ScmServiceInterfaceProxy$2.run(ScmServiceInterfaceProxy.java:168)at com.ibm.team.repository.client.internal.TeamRepository$3.run(TeamRepository.java:1320)at com.ibm.team.repository.common.transport.CancelableCaller.call(CancelableCaller.java:79)at com.ibm.team.repository.client.internal.TeamRepository.callCancelableService(TeamRepository.java:1315)at com.ibm.team.scm.client.internal.ScmClientLibraryContext.callCancelableService(ScmClientLibraryContext.java:71)at com.ibm.team.scm.client.internal.ScmServiceInterfaceProxy.invokeCancelableService(ScmServiceInterfaceProxy.java:164)at com.ibm.team.scm.client.internal.ScmServiceInterfaceProxy.invoke(ScmServiceInterfaceProxy.java:92)at com.sun.proxy.$Proxy22.refreshWorkspaces(Unknown Source)at com.ibm.team.scm.client.internal.WorkspaceManager.getWorkspaceConnection(WorkspaceManager.java:868)at com.ibm.team.build.internal.scm.BuildWorkspaceDescriptor.getConnection(BuildWorkspaceDescriptor.java:165)at com.ibm.team.build.internal.scm.SourceControlUtility.checkForIncoming(SourceControlUtility.java:912)at com.ibm.team.build.internal.engine.JazzScmPreBuildParticipant.shouldBuild(JazzScmPreBuildParticipant.java:87)at com.ibm.team.build.internal.engine.BuildLoop.shouldBuild(BuildLoop.java:1294)at com.ibm.team.build.internal.engine.BuildLoop.runLoop(BuildLoop.java:400)at com.ibm.team.build.internal.engine.BuildLoop$1.run(BuildLoop.java:212)at java.lang.Thread.run(Thread.java:744)Caused by: com.ibm.team.repository.common.LicenseNotGrantedException: Timeout waiting for connectionat com.ibm.team.repository.service.compatibility.license.internal.LicenseService.assertLicenseRemotely(LicenseService.java:168)at com.ibm.team.repository.service.compatibility.license.internal.LicenseService.assertLicense(LicenseService.java:144)at com.ibm.team.repository.service.compatibility.license.internal.LicenseService.assertLicenseForContributor(LicenseService.java:174)at com.ibm.team.repository.service.compatibility.license.internal.LicenseService.assertLicense(LicenseService.java:454)at sun.reflect.GeneratedMethodAccessor141.invoke(null)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)at java.lang.reflect.Method.invoke(Method.java:611)at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)at com.sun.proxy.$Proxy253.assertLicense(null)at com.ibm.team.repository.service.internal.license.jts.JtsLicenseService.assertLicense(JtsLicenseService.java:38)at sun.reflect.GeneratedMethodAccessor140.invoke(null)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)at java.lang.reflect.Method.invoke(Method.java:611)at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)at com.sun.proxy.$Proxy404.assertLicense(null)at com.ibm.team.scm.service.internal.AbstractScmService.assertLicense(AbstractScmService.java:266)... 92 moreCaused by: com.ibm.team.repository.common.TeamRepositoryException: Timeout waiting for connectionat com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClient.doCommunicate(CrossServerHttpClient.java:196)at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClient.send(CrossServerHttpClient.java:447)at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClient.send(CrossServerHttpClient.java:336)at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClient.send(CrossServerHttpClient.java:460)at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClientService.send(CrossServerHttpClientService.java:68)at sun.reflect.GeneratedMethodAccessor123.invoke(null)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)at java.lang.reflect.Method.invoke(Method.java:611)at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)at com.sun.proxy.$Proxy252.send(null)at com.ibm.team.repository.service.compatibility.AbstractCompatibilityService.send(AbstractCompatibilityService.java:56)at com.ibm.team.repository.service.compatibility.license.internal.LicenseService.assertLicenseRemotely(LicenseService.java:165)... 111 moreCaused by: java.io.InterruptedIOException: <Remote exception was of type 'org.apache.http.conn.ConnectionPoolTimeoutException'>Timeout waiting for connectionat org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:417)at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300)at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224)at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:401)at com.ibm.team.jfs.app.http.client.internal.ExtensibleRequestDirector.execute(ExtensibleRequestDirector.java:149)at com.ibm.team.jfs.app.http.client.cache.HttpClientCache.executeCacheMiss(HttpClientCache.java:217)at com.ibm.team.jfs.app.http.client.cache.HttpClientCache.executeCacheMiss(HttpClientCache.java:201)at com.ibm.team.jfs.app.http.client.cache.HttpClientCache.execute(HttpClientCache.java:140)at com.ibm.team.jfs.app.http.client.cache.HttpClientCache.execute(HttpClientCache.java:118)at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:776)at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClient.doCommunicate(CrossServerHttpClient.java:185)... 124 more
Accepted answer
I have filed Build gets stuck in pending if connection to repo fails during SCM pre-build check (309640) for this. Please see my comments there.
One other answer
Based on the error:
com.ibm.team.repository.common.LicenseNotGrantedException: Unable to obtain license to read SCM related data: Timeout waiting for connection.
This seems to indicate a problem accessing jts to validate the user running the builds has the correct authority
I would review the jts logs to see if there were any issues at the time this exception was thrown.
Now if you feel this scenario should be handled more gracefully in RTC, yes open a defect workitem
com.ibm.team.repository.common.LicenseNotGrantedException: Unable to obtain license to read SCM related data: Timeout waiting for connection.
This seems to indicate a problem accessing jts to validate the user running the builds has the correct authority
I would review the jts logs to see if there were any issues at the time this exception was thrown.
Now if you feel this scenario should be handled more gracefully in RTC, yes open a defect workitem
Comments
Scott Cowan
JAZZ DEVELOPER Mar 27 '14, 9:18 a.m.@heatherf I wonder if this exception is a coincidence, or if there's a new case we need to add to our improved retry logic in 4.0.5?
Make JBE's SCM participant resilient to temporary server outages and retryable request failures (265141)