RTC Build Server Recommendations
We are trying to set up a standard build and deployment environment for our developers and for our deployment support team.
What I would like to know is if there are any standards regarding how many build engines should be run on a specific server, things to be concerned about when running multiple build engines on a server, etc?
I have already looked through this:
https://jazz.net/wiki/bin/view/Main/BuildFAQ
And it is very possible we are going abotu this the wrong way (i.e. administerting a number of standard build/deployment servers for all projects/applications.)
We will probably have at elast 2 sets of build servers (one for our DEV and QA environments and one for our PROD environment).
We currently have over 200 project/applications in RTC and want to plan to roll out build and deployment automation for developers in the coming year aggressively.
3 answers
Generally the recommendation is one build engine per machine.
It's possible to configure multiple build engines per machine if the machine has sufficient capacity (disk space & I/O throughput), but you must be careful to keep your builds from trampling all over each other. When I've seen teams configure multiple JBEs on a single machine each JBE instance was confined to it's own little sandbox on the filesystem, though that's impossible to enforce (i.e. sneaky devs can easily violate the isolation) unless you're using something like a chroot jail.
It's possible to configure multiple build engines per machine if the machine has sufficient capacity (disk space & I/O throughput), but you must be careful to keep your builds from trampling all over each other. When I've seen teams configure multiple JBEs on a single machine each JBE instance was confined to it's own little sandbox on the filesystem, though that's impossible to enforce (i.e. sneaky devs can easily violate the isolation) unless you're using something like a chroot jail.
There is no a-priori reason why you couldn't have the same build machines supporting builds for different purposes / environments, but it would mean that the build machines have access to both DEV and QA environments, which you might prefer to isolate.
As for build farm sizing, since JBE can only process one build at a time, you should try to estimate the maximum of how many builds will be running at any given time.
In general we recommend each JBE runs on its own (real or virtual) machine, but it is possible to run multiple JBEs on the same machine if you're careful to avoid builds stomping on each other. See https://jazz.net/wiki/bin/view/Main/BuildFAQ#MultipleJBEs
In our own self-hosting, we have a shared build farm with on the order of 100 machines running JBE, supporting ~500 build definitions across our Jazz Foundation, Rational Team Concert and CALM project areas. Some of these are running on the same machine.
I maintain the group of machines and build engines used to run Jazz Foundation, RTC, and CLM builds (the group Nick refers to above). This consists of 519 build definitions running on 107 build engines on 59 physical machines. The 59 machines span a wide configuration range. The oldest have 2 CPUs and about 4G RAM and we only run one engine on those and that engine is limited to doing "small non-critical builds". The newest have 48 CPUS, 256G RAM, and we're currently running up to 11 engines on those. The number of engines we can run is a product of the resources on the machines and the demands of the specific builds enabled on the engines.
In arriving at our current setup, we have measured a lot of test builds performance time. But also, our product builds are very big, and one trick we've used to speed up builds is to create RAM disks, and do the builds there. Since RAM is expensive, the number of build engines we have on a machine has generally been limited by the amout of RAM we have on that machine.
Considerations when running > 1 engine on a machine include:
o Making sure you don't kill the wrong engines or build processes when "cleaning up" a problem build or engine.
o Being aware of the state of all the engines when rebooting a machine.
o Making sure each engine has its own build area and configuration area.
o Making sure our process for automatically restarting the engines should they fail works properly. (EG if this process is manual, it's easy to lose track of an engine that's down).
o Making sure that engines are configured as uniformly as is practical.
General considerations when managing >> build engines include:
o Load balancing the definitions across the engines so as to reduce pending time for all builds, but giving preference to important builds.
o Steering important builds towards the faster engines (this can be done by using a shorter sleep time for the fast engines than for the slow engines).
o Cleaning up disk space used by the build engines in a way that doesn't delete areas while in use, yet minimizes disk space consumed by prior builds.
As I write all this, it occurs to me I should give a presentation on this topic. I'll add that to my TODO list. Feel free to respond with specific followup questions. They would probably provide me with good direction for my presentation.
In arriving at our current setup, we have measured a lot of test builds performance time. But also, our product builds are very big, and one trick we've used to speed up builds is to create RAM disks, and do the builds there. Since RAM is expensive, the number of build engines we have on a machine has generally been limited by the amout of RAM we have on that machine.
Considerations when running > 1 engine on a machine include:
o Making sure you don't kill the wrong engines or build processes when "cleaning up" a problem build or engine.
o Being aware of the state of all the engines when rebooting a machine.
o Making sure each engine has its own build area and configuration area.
o Making sure our process for automatically restarting the engines should they fail works properly. (EG if this process is manual, it's easy to lose track of an engine that's down).
o Making sure that engines are configured as uniformly as is practical.
General considerations when managing >> build engines include:
o Load balancing the definitions across the engines so as to reduce pending time for all builds, but giving preference to important builds.
o Steering important builds towards the faster engines (this can be done by using a shorter sleep time for the fast engines than for the slow engines).
o Cleaning up disk space used by the build engines in a way that doesn't delete areas while in use, yet minimizes disk space consumed by prior builds.
As I write all this, it occurs to me I should give a presentation on this topic. I'll add that to my TODO list. Feel free to respond with specific followup questions. They would probably provide me with good direction for my presentation.