RTC Build Forge Integration- properties not passed correctly
I posted last week, and was eventually able to resolve some RTC and Build Forge integration issues by upgrading my server plugin to the latest 7.1.1.4 version. At the time that seemed to resolve our issues but we have come to realize that things are still not behaving as they should. There are a number of strange symptoms that seem loosely related.
We have two builds defined in Build Forge that execute properly when started from Build Forge itself. However, the same two builds fail for different (but somewhat related) reasons when started from RTC. It appears the properties are not being correctly mapped from RTC to Build Forge. One build will actually start but fails quickly because the properties are not correct. The second build never actually makes a connection with Build Forge and is abandoned within 3 seconds. The label of that build is also strangely set to the value of one of the properties. I have Build Forge logging enabled on the server but nothing jumps out at me from that log. I do see a failure in the WAS SystemErr.log that may be related: com.ibm.rational.buildforge.team.service.BuildForgeBuildLoopScheduledTask.runTask(BuildForgeBuildLoopScheduledTask.java:45) 0000002f SystemErr R at com.ibm.team.repository.service.async.AbstractAutoScheduledTask.executeTask(AbstractAutoScheduledTask.java:88) 0000002f SystemErr R at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) 0000002f SystemErr R at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 0000002f SystemErr R at java.lang.reflect.Method.invoke(Method.java:618) 0000002f SystemErr R at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370) 0000002f SystemErr R at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356) 0000002f SystemErr R at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56) 0000002f SystemErr R at $Proxy234.executeTask(Unknown Source) 0000002f SystemErr R at com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner.runTask(AsynchronousTaskRunner.java:118) 0000002f SystemErr R at com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner.run(AsynchronousTaskRunner.java:92) 0000002f SystemErr R at java.lang.Thread.run(Thread.java:810) I can provide additional logs if they would be helpful. Is it possible that this is a client/server mismatch issue? Our RTC clients are at 7.1.1.3 and the RTC server plugin is at 7.1.1.4. Pete seemed to think they would work together properly. If this is a version issue, where can I get the 7.1.1.4 client plugin? Thanks for your help, -Dave |
6 answers
|
I found 2 Build Forge services that seem to be stuck in the active services:
com.ibm.rational.buildforge.team.service.BuildForgeEventPollerScheduledTask.runTask(BuildForgeEventPollerScheduledTask.java:34) com.ibm.team.repository.service.async.AbstractAutoScheduledTask.executeTask(AbstractAutoScheduledTask.java:88) sun.reflect.GeneratedMethodAccessor115.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:618) org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370) org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356) org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56) $Proxy234.executeTask(Unknown Source) com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner.runTask(AsynchronousTaskRunner.java:118) com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner.run(AsynchronousTaskRunner.java:92) java.lang.Thread.run(Thread.java:810) com.ibm.jsse2.gc.c(gc.java:174) com.ibm.jsse2.gc.wrap(gc.java:293) javax.net.ssl.SSLEngine.wrap(SSLEngine.java:11) com.buildforge.services.common.ssl.BuildForgeSSLEngine.write(BuildForgeSSLEngine.java:156) com.buildforge.services.client.api.SecureAPIClientBuffer.flush(SecureAPIClientBuffer.java:64) com.buildforge.services.common.api.JsonProtocolImpl.writeFooter(JsonProtocolImpl.java:363) com.buildforge.services.client.api.APIClientConnection.call(APIClientConnection.java:581) com.buildforge.services.client.api.APIClientConnection.translate(APIClientConnection.java:972) com.buildforge.services.client.api.APIClientConnection.translate(APIClientConnection.java:947) com.buildforge.services.client.api.APIClientConnection.render(APIClientConnection.java:980) com.buildforge.services.common.dbo.MessageDBO.translate(MessageDBO.java:118) com.buildforge.services.common.ServiceException.getMessage(ServiceException.java:140) java.lang.Throwable.getLocalizedMessage(Throwable.java:131) java.lang.Throwable.toString(Throwable.java:275) java.lang.Throwable.printStackTrace(Throwable.java:211) java.lang.Throwable.printStackTrace(Throwable.java:163) com.ibm.rational.buildforge.team.service.BuildForgeBuildLoopRunnable.startBuildForgeProject(BuildForgeBuildLoopRunnable.java:204) com.ibm.rational.buildforge.team.service.BuildForgeBuildLoopRunnable.run(BuildForgeBuildLoopRunnable.java:122) com.ibm.team.repository.service.internal.permission.ImpersonationService.runAs(ImpersonationService.java:44) sun.reflect.GeneratedMethodAccessor123.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:618) org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370) org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356) org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56) $Proxy80.runAs(Unknown Source) com.ibm.rational.buildforge.team.service.BuildForgeBuildLoopScheduledTask.runTask(BuildForgeBuildLoopScheduledTask.java:45) com.ibm.team.repository.service.async.AbstractAutoScheduledTask.executeTask(AbstractAutoScheduledTask.java:88) sun.reflect.GeneratedMethodAccessor115.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:618) org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370) org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356) org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56) $Proxy233.executeTask(Unknown Source) com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner.runTask(AsynchronousTaskRunner.java:118) com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner.run(AsynchronousTaskRunner.java:92) java.lang.Thread.run(Thread.java:810) Both have been running for almost 30 minutes. I believe these are supposed to be quick processes. |
Hello Dave,
when you think that this could be a client/server issue ... does this mean that running the BF projects from the RTC server works fine? On the other hand, have you tried "re-importing" the projects definitions in RTC ?, for previous versions of the software, I found that at any change I made to BF projects I had to refresh the definition at RTC build definition to have all configurations refreshed. Regards, Jorge. |
(Dave's actually been posting on my behalf -- let me try to see
what I can add...) First, these are all "new" RTC build defs -- we've deleted and recreated (many times). Yes, the projects in questions all work fine from BF console. I've discovered that it seems to be related to the BF projects having (large?) environments. Today, I tried creating a super-simple test project in BF -- 1 step that visits build machines and executes "hostname". Works fine from BF. Created an RTC build def and it seems to work fine. Since an RTC build of a simple (no-environment) build works, I tried an experiment -- made my simple test build do ".runwait real-project" where real-project is the project I wish I could run from RTC. Lo and behold, the called project works fine (as if we'd run it from BF console.) Mind you, the real project RTC build def is still present so when the build is run from BF by the RTC "build-runner", RTC gets the status updates, step logs, etc. (It's cool that BF initiated build results are tracked in RTC even though not launched from there.) So, my hypothesis is that projects with (big?) environments (we have 30 or so EV's that get picked up as RTC properties) are causing RationalBuildForgeConnector indigestion. This "wrapper-build" approach might be a way forward for us temporarily but it's pretty ugly in since it means our 10-15 BF projects turn into 20-30 with 20-30 corresponding RTC build defs. Users will invariably end up running one of the real builds which at best will crash and burn but more often than not, throws the connector into the "warning" state necessitating a Jazz recycle. Not sure we can do that in a production environment very long... Thanks for any help/insight, Michael Gray Hello Dave, |
Michael, It's not likely due to the large environment, but might be due to some specific property configuration that may not be dealt with properly and it's having trouble at that point. The best thing to do is to gather trace and send it to me to review. I should be able to pinpoint fairly quickly what's going wrong. Please send to pbirk@us.ibm.com and I'll respond on what to do or provide a fix. To enable trace, edit server.startup.bat and add -Dcom.buildforge.rtc.plugin.debug=true to the JAVA_OPTS line. Restart RTC and you should see a buildforge_service.log appear in the startup directory once you submit the first build. Recreate the problem and send the trace to me (you can review it first and there may be something evident in it that you could fix). Then disable the trace again and restart your RTC server again. NOTE: We're making the trace process simpler in RTC 3.0 so it can be dynamically enabled/disabled. Regards, Pete |
|
My trusty Jazz admin Dave will be sending along the logs shortly.
Meanwhile, I've been pressing ahead with my "wrapper project" approach and that seems to work. I've not got my wrapper BF project set up with (an empty) project environment so buildUuid stuff works and I've replaced the ".runwait <proj>" step with ".sleep 0" and "inline <proj>" and set the step environment to the called-project's project environment. That seems to work and now I'm getting step logs etc. flowing back into RTC build results. This has exposed a new problem. Since I had 1 working build through this approach, I created a second one and requested a build, but... The second build is just sitting in the queue, apparently waiting for the (2-3 hour) running build to complete. Please tell me the BuildForgeConnector Engine isn't a one-at-a-time thing?! We really need to be able to run multiple BF builds in parallel... Do we need multiple BuildForgeConnector Engine instances or something?
Michael, It's not likely due to the large environment, but might be due to some specific property configuration that may not be dealt with properly and it's having trouble at that point. The best thing to do is to gather trace and send it to me to review. I should be able to pinpoint fairly quickly what's going wrong. Please send to pbirk@us.ibm.com and I'll respond on what to do or provide a fix. To enable trace, edit server.startup.bat and add -Dcom.buildforge.rtc.plugin.debug=true to the JAVA_OPTS line. Restart RTC and you should see a buildforge_service.log appear in the startup directory once you submit the first build. Recreate the problem and send the trace to me (you can review it first and there may be something evident in it that you could fix). Then disable the trace again and restart your RTC server again. NOTE: We're making the trace process simpler in RTC 3.0 so it can be dynamically enabled/disabled. Regards, Pete |
|
The APPEND and PREPEND variable types are not currently supported by the Build Forge RTC plugin. This is being address via RTC enhancement 117134.
Pete |
Your answer
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.