It's all about the answers!

Ask a question

build engine doesn't pickup pending builds


Stefan Stern (406128) | asked Mar 11 '07, 5:00 a.m.
Hello.
On Win2k, which we use at Bosch as OS on almost all computers, we had
the problem that the build engine wouldn't start on the pending builds.
Instead, the build engine remains idle while the build queue holds one
or more pending builds.
We then installed a Windows XP host, running the server on it, starting
the build engine there and injected new builds via our Win2k machines
where we ran the Jazz client. All worked fine.
Now I tried to run server and client on one XP host. I tried for, hm,
about 2 hours, but my build engine is not willing to start on my pending
builds.
I checked the JDKs/JREs, I checked the environment variables, I redid
the build definition and build engine setup both in following exactly to
the tutorials and by adapting folders and settings to for instance the
IBM JRE coming with the server instead of the Sun JDK. All without success.
I saw only one issue, telling that a build engine would not pickup a
pending build when the engine was disabled for that build definition
when the build was requested and was re-enabled for it afterwards.
I don't think that this is the case in my setup here.
I deactivated virus scanner, firewall, etc. Had no influence (in case
the communication is blocked when the engine tries to check for pending
builds).
Anyone an idea? I will keep on testing. Maybe I can get it running when
invoking the server and build engine on another host as the Jazz client.

Nice sunday to all of you, who read this the day I wrote it. Before I
continue on testing, I will get another cup of coffee. Maybe it helps,
surely it won't hurt! :-)

Regards,
Stefan

7 answers



permanent link
Stefan Stern (406128) | answered Mar 15 '07, 1:53 p.m.
Hello

We are a little confused why you are running out of memory.

So was I. :-) I haven't found the reason yet. Instead, I'm running out
of ideas what might be the problem. I will reduce my interval to 30 min.
But I don't think that this will solve the problem. I can install
JDK5_11, currently we use Suns JDK5_10.

Based on your post...it seems like you didn't have a complicated
build that was being invoked every ten minutes...is that true? I'd
like to try to simulate what you did as closely as I can in hopes of
reproducing.


No, the build is trivial. It almost does nothing. Please see the
attached files in #19590. It's a screenshot of the build definition, the
build.xml and another screenshot shows the multiple forked JVM instances
running without closing properly. As I commented in #19590, the build
works fine for the first hours. After a while the JVM seems to fork and
not to close the child processes. Could this be an ANT bug? I could use
the latest ANT release instead of the 6.5 version coming with the build
system.

Regards,
Stefan

permanent link
Don Weinand (7851) | answered Mar 14 '07, 11:42 p.m.
JAZZ DEVELOPER
We are a little confused why you are running out of memory. We run 8 build
servers to do all(continuous, nightly, weekly, etc.) Jazz builds. We only
do our continous builds every 30 minutes but we have about 20 build
definitions that all the build servers support.. They usually run for
quite some time without issue. I don't think I've ever seen the build
engine run out of memory on them before. I'm going to set up a continuous
build and let it run over night building in very short intervals to try to
reproduce your problem. Based on your post...it seems like you didn't have
a complicated build that was being invoked every ten minutes...is that true?
I'd like to try to simulate what you did as closely as I can in hopes of
reproducing.

"Stefan Stern" <Stefan.Stern3@de.bosch.com> wrote in message
news:et9t7u$fnl$1@localhost.localdomain...
I'll report back when I got results.

From command line, the engine works fine for at least some hours. I set up
a build definition requesting a build every 10 minutes. I had the engine
run over night and the next morning it reporter OutOfMemoryErrors. I
created a work item for this (see #19590)

But the builds are executed properly. Our next step is to really compile
and test some standard Java projects.

Then we will do the same for some plugin project and a feature, using the
PDE build; again setting up compilation and afterwards the testing.

If anyone has some hints regarding PDE build usage in Jazz (for instance
ordering the source code into /plugins and /features folders), do not
hesitate :-)

Regards,
Stefan

permanent link
Stefan Stern (406128) | answered Mar 14 '07, 6:40 p.m.
I'll report back when I got results.

From command line, the engine works fine for at least some hours. I set
up a build definition requesting a build every 10 minutes. I had the
engine run over night and the next morning it reporter
OutOfMemoryErrors. I created a work item for this (see #19590)

But the builds are executed properly. Our next step is to really compile
and test some standard Java projects.

Then we will do the same for some plugin project and a feature, using
the PDE build; again setting up compilation and afterwards the testing.

If anyone has some hints regarding PDE build usage in Jazz (for instance
ordering the source code into /plugins and /features folders), do not
hesitate :-)

Regards,
Stefan

permanent link
Stefan Stern (406128) | answered Mar 12 '07, 1:46 p.m.
Hello Ryan,

Is it possible that the build engine wasn't really running in
eclipse?

No, the engine was up and running, the "terminate" button was beaming at
me in brightest red. I switched it off and restarted it several times
with different settings like other JREs (Java5 and IBM JRE), canceled
the builds in queue and requested new builds. The engine was running,
telling me that it waited for incoming requests while the queue showed
me 1+ pending requests. The identifiers (build definition, engine label,
etc) matched. The engine should have started the build.

Once you are past hello world, you should be running the engine from
the command line and starting/stopping the engine when it makes sense
for you.

OK. We shouldn't loose time in seeking wrong behavior for a scenario
never meant for real usage. I will setup the command line engine (maybe
I can arrange it as Windows service or mount it in a Linux VM) and then
give it another try. :) I'll report back when I got results.

Regards,
Stefan

permanent link
Ryan Manwiller (1.3k1) | answered Mar 12 '07, 12:45 p.m.
JAZZ DEVELOPER
Hi Stefan,

Is it possible that the build engine wasn't really running in eclipse? Once
you shutdown eclipse, the local test engine is stopped as well. Starting
eclipse back up does not start the build engine again. You can manually
start the engine by running the Jazz Build Engine launch from the Run...
menu in the java perspective. Or, as you've noticed, when you create a new
definition the engine is started for you.

The key point to understand is that the engine is only being launched from
eclipse to help you see a green hello world build as soon as possible. Once
you are past hello world, you should be running the engine from the command
line and starting/stopping the engine when it makes sense for you.

Ryan

permanent link
Stefan Stern (406128) | answered Mar 11 '07, 9:23 a.m.
OK,
running the build engine from the command line instead of running it
from Eclipse works just fine. It picks up my requested builds and runs
through.

Stefan

permanent link
Stefan Stern (406128) | answered Mar 11 '07, 7:47 a.m.
I let the engine run in the background and after an hour or so, there
was this stacktrace in the console:

10:58:17,734 ERROR
..team.repository.service.feed.internal.FeedService - Exception in
FeedService GET :
ClientAbortException: java.net.SocketException: Software caused
connection abort: socket write error
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:366)
at
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433)
at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:398)
at
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:381)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:88)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:82)
at
com.ibm.team.repository.service.feed.internal.FeedService.perform_GET(FeedService.java:536)
at
com.ibm.team.repository.service.feed.internal.FeedService.service(FeedService.java:270)
at
com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.doRestService(AbstractTeamServerServlet.java:482)
at
com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:294)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:129)
at
org.eclipse.equinox.servlet.bridge.http.internal.ServletRegistration.handleRequest(ServletRegistration.java:76)
at
org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.processAlias(ProxyServlet.java:107)
at
org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.service(ProxyServlet.java:73)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

at
org.eclipse.equinox.servlet.bridge.launcher.BridgeServlet.service(BridgeServlet.java:95)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:803)
Caused by:
java.net.SocketException: Software caused connection abort: socket write
error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:105)
at java.net.SocketOutputStream.write(SocketOutputStream.java:149)
at
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:746)
at
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:348)
at
org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:769)
at
org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:117)
at
org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:579)
at org.apache.coyote.Response.doWrite(Response.java:559)
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:361)
... 32 more
10:58:17,828 WARN
common.transport.servlet.AbstractTeamServerServlet - RuntimeException
processing request for GET com.ibm.team.reposi
tory.service.feed.IFeedService for: {UnknownUserid}
java.lang.IllegalStateException
at
org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:404)
at
com.ibm.team.repository.service.feed.internal.FeedService.perform_GET(FeedService.java:554)
at
com.ibm.team.repository.service.feed.internal.FeedService.service(FeedService.java:270)
at
com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.doRestService(AbstractTeamServerServlet.java:482)
at
com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:294)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:129)
at
org.eclipse.equinox.servlet.bridge.http.internal.ServletRegistration.handleRequest(ServletRegistration.java:76)
at
org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.processAlias(ProxyServlet.java:107)
at
org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.service(ProxyServlet.java:73)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.eclipse.equinox.servlet.bridge.launcher.BridgeServlet.service(BridgeServlet.java:95)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:803)
12:07:23,656 ERROR
..team.repository.service.feed.internal.FeedService - Exception in
FeedService GET :
ClientAbortException: java.net.SocketException: Software caused
connection abort: socket write error
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:366)
at
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433)
at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:398)
at
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:381)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:88)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:82)
at
com.ibm.team.repository.service.feed.internal.FeedService.perform_GET(FeedService.java:536)
at
com.ibm.team.repository.service.feed.internal.FeedService.service(FeedService.java:270)
at
com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.doRestService(AbstractTeamServerServlet.java:482)
at
com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:294)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

at
org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:129)
at
org.eclipse.equinox.servlet.bridge.http.internal.ServletRegistration.handleRequest(ServletRegistration.java:76)
at
org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.processAlias(ProxyServlet.java:107)
at
org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.service(ProxyServlet.java:73)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.eclipse.equinox.servlet.bridge.launcher.BridgeServlet.service(BridgeServlet.java:95)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:803)
Caused by:
java.net.SocketException: Software caused connection abort: socket write
error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:105)
at java.net.SocketOutputStream.write(SocketOutputStream.java:149)
at
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:746)
at
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:348)
at
org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:769)
at
org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:117)
at
org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:579)
at org.apache.coyote.Response.doWrite(Response.java:559)
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:361)
... 32 more
12:07:23,671 WARN
common.transport.servlet.AbstractTeam
ServerServlet - RuntimeException processing request for GET
com.ibm.team.reposi
tory.service.feed.IFeedService for: {UnknownUserid}
java.lang.IllegalStateException
at
org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:404)
at
com.ibm.team.repository.service.feed.internal.FeedService.perform_GET(FeedService.java:554)
at
com.ibm.team.repository.service.feed.internal.FeedService.service(FeedService.java:270)
at
com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.doRestService(AbstractTeamServerServlet.java:482)
at
com.ibm.team.repository.common.transport.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:294)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

at
org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:129)

at
org.eclipse.equinox.servlet.bridge.http.internal.ServletRegistration.handleRequest(ServletRegistration.java:76)
at
org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.processAlias(ProxyServlet.java:107)
at
org.eclipse.equinox.servlet.bridge.http.internal.ProxyServlet.service(ProxyServlet.java:73)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

at
org.eclipse.equinox.servlet.bridge.launcher.BridgeServlet.service(BridgeServlet.java:95)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:803)

Your answer


Register or to post your answer.