I maintain the group of machines and build engines used to run Jazz Foundation, RTC, and CLM builds (the group Nick refers to above).  This consists of 519 build definitions running on 107 build engines on 59 physical machines.  The 59 machines span a wide configuration range.  The oldest have 2 CPUs and about 4G RAM and we only run one engine on those and that engine is limited to doing "small non-critical builds".  The newest have 48 CPUS, 256G RAM, and we're currently running up to 11 engines on those.  The number of engines we can run is a product of the resources on the machines and the demands of the specific builds enabled on the engines.

In arriving at our current setup, we have measured a lot of test builds performance time.  But also, our product builds are very big, and one trick we've used to speed up builds is to create RAM disks, and do the builds there.  Since RAM is expensive, the number of build engines we have on a machine has generally been limited by the amout of RAM we have on that machine.

Considerations when running > 1 engine on a machine include:
  o Making sure you don't kill the wrong engines or build processes when "cleaning up" a problem build or engine.
  o Being aware of the state of all the engines when rebooting a machine.
  o Making sure each engine has its own build area and configuration area.
  o Making sure our process for automatically restarting the engines should they fail works properly.  (EG if this process is manual, it's easy to lose track of an engine that's down).
  o Making sure that engines are configured as uniformly as is practical.

General considerations when managing >> build engines include:
  o Load balancing the definitions across the engines so as to reduce pending time for all builds, but giving preference to important builds.
  o Steering important builds towards the faster engines (this can be done by using a shorter sleep time for the fast engines than for the slow engines).
  o Cleaning up disk space used by the build engines in a way that doesn't delete areas while in use, yet minimizes disk space consumed by prior builds.

As I write all this, it occurs to me I should give a presentation on this topic.  I'll add that to my TODO list.  Feel free to respond with specific followup questions.  They would probably provide me with good direction for my presentation.

There is no a-priori reason why you couldn't have the same build machines supporting builds for different purposes / environments, but it would mean that the build machines have access to both DEV and QA environments, which you might prefer to isolate. As for build farm sizing, since JBE can only process one build at a time, you should try to estimate the maximum of how many builds will be running at any given time.

Generally the recommendation is one build engine per machine.

It's possible to configure multiple build engines per machine if the machine has sufficient capacity (disk space & I/O throughput), but you must be careful to keep your builds from trampling all over each other. When I've seen teams configure multiple JBEs on a single machine each JBE instance was confined to it's own little sandbox on the filesystem, though that's impossible to enforce (i.e. sneaky devs can easily violate the isolation) unless you're using something like a chroot jail.