build engine doesn't pickup pending builds
Hello.
On Win2k, which we use at Bosch as OS on almost all computers, we had the problem that the build engine wouldn't start on the pending builds. Instead, the build engine remains idle while the build queue holds one or more pending builds. We then installed a Windows XP host, running the server on it, starting the build engine there and injected new builds via our Win2k machines where we ran the Jazz client. All worked fine. Now I tried to run server and client on one XP host. I tried for, hm, about 2 hours, but my build engine is not willing to start on my pending builds. I checked the JDKs/JREs, I checked the environment variables, I redid the build definition and build engine setup both in following exactly to the tutorials and by adapting folders and settings to for instance the IBM JRE coming with the server instead of the Sun JDK. All without success. I saw only one issue, telling that a build engine would not pickup a pending build when the engine was disabled for that build definition when the build was requested and was re-enabled for it afterwards. I don't think that this is the case in my setup here. I deactivated virus scanner, firewall, etc. Had no influence (in case the communication is blocked when the engine tries to check for pending builds). Anyone an idea? I will keep on testing. Maybe I can get it running when invoking the server and build engine on another host as the Jazz client. Nice sunday to all of you, who read this the day I wrote it. Before I continue on testing, I will get another cup of coffee. Maybe it helps, surely it won't hurt! :-) Regards, Stefan |
7 answers
|
I let the engine run in the background and after an hour or so, there
was this stacktrace in the console: 10:58:17,734 ERROR ..team.repository.service.feed.internal.FeedService - Exception in FeedService GET : ClientAbortException: java.net.SocketException: Software caused connection abort: socket write error at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:366) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433) at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:398) at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:381) at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:88) at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:82) at com.ibm.team.repository.service.feed.internal.FeedService.perform_GET(FeedService.java:536) at com.ibm.team.repository.service.feed.internal.FeedService.service(FeedService.java:270) at com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.doRestService(AbstractTeamServerServlet.java:482) at com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:294) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:129) at org.eclipse.equinox.servlet.bridge.http.internal.ServletRegistration.handleRequest(ServletRegistration.java:76) at org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.processAlias(ProxyServlet.java:107) at org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.service(ProxyServlet.java:73) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.eclipse.equinox.servlet.bridge.launcher.BridgeServlet.service(BridgeServlet.java:95) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:803) Caused by: java.net.SocketException: Software caused connection abort: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:105) at java.net.SocketOutputStream.write(SocketOutputStream.java:149) at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:746) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433) at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:348) at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:769) at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:117) at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:579) at org.apache.coyote.Response.doWrite(Response.java:559) at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:361) ... 32 more 10:58:17,828 WARN common.transport.servlet.AbstractTeamServerServlet - RuntimeException processing request for GET com.ibm.team.reposi tory.service.feed.IFeedService for: {UnknownUserid} java.lang.IllegalStateException at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:404) at com.ibm.team.repository.service.feed.internal.FeedService.perform_GET(FeedService.java:554) at com.ibm.team.repository.service.feed.internal.FeedService.service(FeedService.java:270) at com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.doRestService(AbstractTeamServerServlet.java:482) at com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:294) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:129) at org.eclipse.equinox.servlet.bridge.http.internal.ServletRegistration.handleRequest(ServletRegistration.java:76) at org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.processAlias(ProxyServlet.java:107) at org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.service(ProxyServlet.java:73) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.eclipse.equinox.servlet.bridge.launcher.BridgeServlet.service(BridgeServlet.java:95) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:803) 12:07:23,656 ERROR ..team.repository.service.feed.internal.FeedService - Exception in FeedService GET : ClientAbortException: java.net.SocketException: Software caused connection abort: socket write error at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:366) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433) at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:398) at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:381) at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:88) at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:82) at com.ibm.team.repository.service.feed.internal.FeedService.perform_GET(FeedService.java:536) at com.ibm.team.repository.service.feed.internal.FeedService.service(FeedService.java:270) at com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.doRestService(AbstractTeamServerServlet.java:482) at com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:294) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:129) at org.eclipse.equinox.servlet.bridge.http.internal.ServletRegistration.handleRequest(ServletRegistration.java:76) at org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.processAlias(ProxyServlet.java:107) at org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.service(ProxyServlet.java:73) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.eclipse.equinox.servlet.bridge.launcher.BridgeServlet.service(BridgeServlet.java:95) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:803) Caused by: java.net.SocketException: Software caused connection abort: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:105) at java.net.SocketOutputStream.write(SocketOutputStream.java:149) at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:746) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433) at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:348) at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:769) at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:117) at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:579) at org.apache.coyote.Response.doWrite(Response.java:559) at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:361) ... 32 more 12:07:23,671 WARN common.transport.servlet.AbstractTeam ServerServlet - RuntimeException processing request for GET com.ibm.team.reposi tory.service.feed.IFeedService for: {UnknownUserid} java.lang.IllegalStateException at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:404) at com.ibm.team.repository.service.feed.internal.FeedService.perform_GET(FeedService.java:554) at com.ibm.team.repository.service.feed.internal.FeedService.service(FeedService.java:270) at com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.doRestService(AbstractTeamServerServlet.java:482) at com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:294) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:129) at org.eclipse.equinox.servlet.bridge.http.internal.ServletRegistration.handleRequest(ServletRegistration.java:76) at org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.processAlias(ProxyServlet.java:107) at org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.service(ProxyServlet.java:73) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.eclipse.equinox.servlet.bridge.launcher.BridgeServlet.service(BridgeServlet.java:95) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:803) |
|
OK,
running the build engine from the command line instead of running it from Eclipse works just fine. It picks up my requested builds and runs through. Stefan |
Hi Stefan,
Is it possible that the build engine wasn't really running in eclipse? Once you shutdown eclipse, the local test engine is stopped as well. Starting eclipse back up does not start the build engine again. You can manually start the engine by running the Jazz Build Engine launch from the Run... menu in the java perspective. Or, as you've noticed, when you create a new definition the engine is started for you. The key point to understand is that the engine is only being launched from eclipse to help you see a green hello world build as soon as possible. Once you are past hello world, you should be running the engine from the command line and starting/stopping the engine when it makes sense for you. Ryan |
|
Hello Ryan,
Is it possible that the build engine wasn't really running in No, the engine was up and running, the "terminate" button was beaming at me in brightest red. I switched it off and restarted it several times with different settings like other JREs (Java5 and IBM JRE), canceled the builds in queue and requested new builds. The engine was running, telling me that it waited for incoming requests while the queue showed me 1+ pending requests. The identifiers (build definition, engine label, etc) matched. The engine should have started the build. Once you are past hello world, you should be running the engine from OK. We shouldn't loose time in seeking wrong behavior for a scenario never meant for real usage. I will setup the command line engine (maybe I can arrange it as Windows service or mount it in a Linux VM) and then give it another try. :) I'll report back when I got results. Regards, Stefan |
I'll report back when I got results. From command line, the engine works fine for at least some hours. I set up a build definition requesting a build every 10 minutes. I had the engine run over night and the next morning it reporter OutOfMemoryErrors. I created a work item for this (see #19590) But the builds are executed properly. Our next step is to really compile and test some standard Java projects. Then we will do the same for some plugin project and a feature, using the PDE build; again setting up compilation and afterwards the testing. If anyone has some hints regarding PDE build usage in Jazz (for instance ordering the source code into /plugins and /features folders), do not hesitate :-) Regards, Stefan |
We are a little confused why you are running out of memory. We run 8 build
servers to do all(continuous, nightly, weekly, etc.) Jazz builds. We only do our continous builds every 30 minutes but we have about 20 build definitions that all the build servers support.. They usually run for quite some time without issue. I don't think I've ever seen the build engine run out of memory on them before. I'm going to set up a continuous build and let it run over night building in very short intervals to try to reproduce your problem. Based on your post...it seems like you didn't have a complicated build that was being invoked every ten minutes...is that true? I'd like to try to simulate what you did as closely as I can in hopes of reproducing. "Stefan Stern" <Stefan.Stern3@de.bosch.com> wrote in message news:et9t7u$fnl$1@localhost.localdomain... I'll report back when I got results. |
|
Hello
We are a little confused why you are running out of memory. So was I. :-) I haven't found the reason yet. Instead, I'm running out of ideas what might be the problem. I will reduce my interval to 30 min. But I don't think that this will solve the problem. I can install JDK5_11, currently we use Suns JDK5_10. Based on your post...it seems like you didn't have a complicated No, the build is trivial. It almost does nothing. Please see the attached files in #19590. It's a screenshot of the build definition, the build.xml and another screenshot shows the multiple forked JVM instances running without closing properly. As I commented in #19590, the build works fine for the first hours. After a while the JVM seems to fork and not to close the child processes. Could this be an ANT bug? I could use the latest ANT release instead of the 6.5 version coming with the build system. Regards, Stefan |
Your answer
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.