It's all about the answers!

Ask a question

RTC build abnormally slow compared to local builds


Jean-Michel Athane (671112) | asked Jan 21 '11, 4:45 a.m.
I setup C++ continuous integration for a customer. The build is a classic Makefile build. So I use a command line build (with gmake). When I use the RTC build engine, each compilation is traced in the build result.
When I build the application locally in my Eclipse workspace it goes very fast. less than one second for a compilation.
I test my production model on a VMWare which is RTC client, server and build server. When I build with the RTC build engine, the RTC build log shows that each compilation takes 4 seconds.
On the real execution environment (client, server and build server are different machines), the RTC build log shows that each compilation takes 14 seconds. The ping between the different machines is less than 1ms.
The current version of RTC is the latest 2 version. We plan to migrate to V3 soon.
so my questions are :
- did you already see such performance degradations ?
- do you have one idea of the reason (the java calls for tracing the activities ? the way the log is handled ?) ? Any workaround ?
- do you think that a migration to V3 will improve the situation ?

5 answers



permanent link
Nick Edgar (6.5k711) | answered Jan 21 '11, 3:59 p.m.
JAZZ DEVELOPER
Jean-Michel,

About the only expected performance difference is in the time it takes JBE before it asks for the next build to process. By default it polls every 30 seconds, but that can be adjusted via the -sleepTime argument.
But it sounds like you're already taking that into account.

There are also some known scalability issues, e.g. if the build reports many build activities (i.e. 1000s), but it doesn't sound like you're doing that either. But if you're calling out to Ant from the command line build script to report activities or publish logs or other data to the build result, the overhead of this could be significant.

Are the timings you give just the time to invoke gmake, or do they include some of the above items?

Is the whole output from gmake captured in the build log? If so, maybe there's a problem in how efficiently we do this. Could you try redirecting its output to a different file to see if that makes any difference?

Regarding RTC v3, there have been some performance improvements throughout, but nothing that would significantly impact this case. It's worth trying, though.

permanent link
Jean-Michel Athane (671112) | answered Jan 26 '11, 10:28 a.m.
Hello Nick,
I did a test on my test VMWare. 82 activities are traced in the build results with java calls (mainly compilations). The build takes 7mins 45sec. If I remove the java calls, it takes .. 11 seconds. The overhead of the java calls is terrible .. How could we change that ???

permanent link
Nick Edgar (6.5k711) | answered Jan 26 '11, 4:14 p.m.
JAZZ DEVELOPER
So, just to confirm, you're saying that the overhead is attributable to the Ant invocations from the makefile to report activities to the build?

Rather than calling out to Ant from your makefile to report activities, one approach would be to have Ant drive the makefile. But then you'd be limited in terms of the granularity of activity reporting.

Although we don't have a separate client program to drive this, other than the Ant tasks, one possibility for activity reporting is to use the REST interface that supports the Build web UI (among other uses).

For example, this script publishes 100 activities:

#!/bin/sh
REPO=https://localhost:9443/jazz
USER_ID=ADMIN
PASSWORD=ADMIN
RESULT_UUID=$1

function auth {
curl -k -L -b cookies -c cookies -s -S -o identity.txt $REPO/authenticated/identity
curl -k -L -b cookies -c cookies -s -S -o security_check.txt -d j_username=$USER_ID -d j_password=$PASSWORD $REPO/j_security_check
}

function start_activity {
curl -k -L -b cookies -c cookies -s -S -o activity.txt -H "Content-Type: text/json" -d "{\"buildResultItemId\": \"$RESULT_UUID\", \"label\": \"$1\", \"autoComplete\":\"true\"}" $REPO/resource/virtual/build/result/$RESULT_UUID/activities
}

rm cookies
date
echo "publishing activities for build result uuid '$RESULT_UUID'"
auth
for ((i=1; i<=100; i++)) do
start_activity "activity $i"
done
date


It's invoked from a command line build using:

/Users/nedgar/Work/publish_activities.sh ${buildResultUUID}


It takes ~14 seconds on my machine (MacBook Pro) talking to a local Jetty server. The output is, e.g.:

2011-01-26 16:06:23 [Jazz build engine]
2011-01-26 16:06:23 [Jazz build engine] Substituted the following configuration element property variables:
2011-01-26 16:06:23 [Jazz build engine] com.ibm.team.build.cmdline : com.ibm.team.build.cmdline.command = /Users/nedgar/Work/publish_activities.sh ${buildResultUUID} --> com.ibm.team.build.cmdline.command = /Users/nedgar/Work/publish_activities.sh _H0BFECmQEeC5FdHNeLKebQ
2011-01-26 16:06:23 [Jazz build engine]
2011-01-26 16:06:23 [Jazz build engine] running on host: Nick-Edgars-MacBook-Pro.local
2011-01-26 16:06:23 [Jazz build engine] Should build occur?
2011-01-26 16:06:23 [Jazz build engine] Yes: Always build a user initiated request.
2011-01-26 16:06:23 [Jazz build engine] Invoking build participant "com.ibm.team.build.cmdline"
Wed Jan 26 16:06:23 EST 2011
publishing activities for build result uuid '_H0BFECmQEeC5FdHNeLKebQ'
Wed Jan 26 16:06:37 EST 2011


One downside of this approach is that it would be hard to nest activities, since there's no easy way to get the parent activity id back. The script would need to capture the response and parse the JSON.

permanent link
Jean-Michel Athane (671112) | answered Feb 08 '11, 1:35 p.m.
GREAT !
Using your code, I could write a script called gmakeLaunch.sh which does the "auth", then calls gmake, which itself calls - when necessary - an other script that does the "start_activity".
The build time on my VMWare is now 25 seconds (it was 8 minutes when I used the java calls for tracing the activities).

Now, I miss only two things :
- How to avoid having the password in the script ..
- what is the syntax of the cURL call for the function publishFile (with the 2 parameters Label and Path).

If you have the info (or some pointers) I would be completely happy (and my customer too !)

Thanks a lot again !

permanent link
Nick Edgar (6.5k711) | answered Feb 08 '11, 3:39 p.m.
JAZZ DEVELOPER
Good to hear you were able to get it working.

For hiding the password, I suggest storing it in a separate file with appropriate permissions, then use the shell's backquote support for substituting the result of a cat command.
e.g.
echo secret >pw.txt
echo `cat pw.txt`
-> secret

For uploading files to the build result, unfortunately that's really not straightforward. The current REST interface requires that you post the content separately to the content service, then pass the resulting location to the Build URL for posting artifact contributions. The content service requires that a hash of the content be computed client-side and passed as a query arg.

As a result, I suggest sticking with the Ant tasks for publishing files. Hopefully the number of invocations here is small enough that it won't adversely affect your build time.

Your answer


Register or to post your answer.