One piece of data that would be important would be to time how long your
Ant build takes when run directly (not by the jbe). If the execution
time of the Ant-only build is the bulk of the execution time, then it is
a pure Ant issue. If it isn't, then we'd want to look at both the
"load" time (when it is updating files in the build sandbox) and
"reporting" time (when it is reporting both intermediate and final build
results back to the RTC server).

Cheers,
Geoff

amcohen.us.ibm.com wrote:

As my project grows bigger and more complicated, I am noticing that
the builds are taking exponentially longer. My project is a mix of
Java and javascript (With Dojo). I am currently using the Jazz Build
Engine with ANT. What are some best practices to make the build go
faster?

Thanks!

--
Aaron

Hi Aaron,

I think the first step should be try to get a handle on where the time is going. Is the majority of the time in JBE for the Jazz SCM accept/load pre-build steps, or is in the invocation of your Ant script?

If it's in the script, can you narrow down where the time is going?

If you're not already doing so, try sprinkling invocations of our startBuildActivity/completeBuildActivity tasks throughout your script to get an idea of where the time is going. The activities show up in the Activies page of the build result editor.

Also note that some built-in Ant tasks like copy do not scale well. This is an issue for our own NL builds where it has to copy many files. It may be better to run a command line copy instead.

Some other things you can try:
- avoid deleting the load directory to improve fetch time
- (advanced) use the requestTeamBuild and waitForTeamBuild tasks to split up your build into sub-builds

Regards,
Nick

From Simon Wiest, a German consultant for build systems:
1. remove. Don's create JavaDoc etc. in a continuous build.
2. cascade, fail early. When the compile fails, don't run the tests.
3. modularize. Reuse artifacts from previous builds that are not
affected by changes. Build only what has changed and what depends on that.
4. parallelize. Compile and test independent modules in parallel.

4. is somewhat interesting because I have not figured out how to have
multiple build servers in RTC. IBM does not sell build server licenses,
at least they are not listed online. I would rather use Ant to launch
sub-builds (that logically belong together) on other machines.

Frank.


amcohen.us.ibm.com schrieb:

As my project grows bigger and more complicated, I am noticing that
the builds are taking exponentially longer. My project is a mix of
Java and javascript (With Dojo). I am currently using the Jazz Build
Engine with ANT. What are some best practices to make the build go
faster?

Thanks!

--
Aaron

--
Dr. Frank Gerhardt

Gerhardt Informatics Kft.
Eclipse Foundation member

fg@gerhardtinformatics.com <mailto>
www.gerhardtinformatics.com <http>

SENS-Experte www.SoftwareExperts.de <http>


Adam u. 9, 6753 Szeged, Hungary. Registergericht/Domicile and Court of
Registry: Szeged, Hungary. HRB-Nr./Commercial Registery No.:
06-09-010204. Umsatzsteuer-Id/VAT registration number: HU 13592417.
Geschftsfhrer/Managing Directors: Borbala Gerhardt, Szilvia Toth
Kontakt in Deutschland: +49-177-757 8658, Orplidstr. 17, 70597 Stuttgart

We are experiencing similar slowness, although I wouldn't classify it as excessive, just not as fast as I would like it. We have some java and lots of javascript + dojo.

For us, 98% of the time is going to fetching the workspace. We don't delete it each time (if we do that it takes more time).

I believe it may have something to do with having zillions of tiny files (dojo distribution).

Frank, there shouldn't be any licensing issues with running multiple build engines, since typically these would all run under the same build user, who is granted the build license.

Note that the build engine acts as a client to the Jazz server, and polls for requests -- it's not itself a server.

For parallelizing builds, you can use the requestTeamBuild and waitForTeamBuild tasks as mentioned above. For example, say you want to split build A into parallel sub-builds B and C. You would create build definitions for A,B,C. The script for A would do requestTeamBuild(B), requestTeamBuild(C),waitForTeamBuild(B),waitForTeamBuild(C). This only works if B and C can run on a different build engine than A, since the first engine will be held up running A.

So for example, if you have 3 build engines, you could assign:
A -> engine 1
B -> engine 2
C -> engine 3

or
A -> engine 1
B -> engine 2,3
C -> engine 2,3

or even
A -> engine 1,2,3
B -> engine 1,2,3
C -> engine 1,2,3

We use this approach for our own integration builds, where each component's tests are run as sub-builds, along with other phases like mixing in translations and creating the installs.

Note, however, that our support for such nested builds is currently pretty weak, since there's no awareness of nesting between the final build results. For example, abandoning A would not automatically abandon B and C. There's also no way of browsing from a parent build to child builds, or vice versa. This is a possible enhancement for 3.0.

For parallelizing builds, you can use the requestTeamBuild and waitForTeamBuild tasks as mentioned above. For example, say you want to split build A into parallel sub-builds B and C. You would create build definitions for A,B,C. The script for A would do requestTeamBuild(B), requestTeamBuild(C),waitForTeamBuild(B),waitForTeamBuild(C). This only works if B and C can run on a different build engine than A, since the first engine will be held up running A.

So for example, if you have 3 build engines, you could assign:
A -> engine 1
B -> engine 2
C -> engine 3

We use this approach for our own integration builds, where each component's tests are run as sub-builds, along with other phases like mixing in translations and creating the installs.

Hi nedgar,
I'm using the Jazz Build Engine 2.0 and there is a parallel build scenario as below,
I want build our service on engine1, test it on engine 2(linux) and engine 3(windows). After build on engine 1, the binary package jar files will be shared on a ftp. So I want to share the path in the ftp so that the engine 2 and 3 can download binary files and then start to test.

My first question is can we share some variable between different build engines?(like pass some parameter to the requestTeamBuild task) The buildLabel is generated by each build engine and the values(seems like the build time) are not identical.

And the second question is how can we get the buildLabel property in ant task which is launched by a command line build template? (If the ant task is launched by a ant build template, I can get the property via ${buildLabel})

For passing parameters between builds, you can specify build properties to add/change/delete when issuing the request via the requestTeamBuild task's overridePropertiesFile and deletePropertiesFile attributes. The referenced files are in Java .properties file format.

For example, to pass the original build's UUID and label, and other properties, to the test build:

<echo message="originalBuildResultUUID=${buildResultUUID}" file="override.properties"/>

<echo message="originalBuildLabel="${buildLabel}" file="override.properties" append="true"/>

<echo message="ftpUrl=ftp://some.url/path" file="override.properties" append="true"/>

<requestTeamBuild 

  repositoryAddress="${repositoryAddress}"

  buildDefinitionId="test"

  overridePropertiesFile="override.properties"/>

To change the build label, you can use the buildResultPublisher task.
For example, in the test build's script you could do:

<buildResultPublisher

  repositoryAddress="${repositoryAddress}"

  buildResultUUID="${buildResultUUID}"

  label="${originalBuildLabel}-test"/>

For more details on these options, see the Build Toolkit Ant Task Reference in the Help (in the Eclipse client).