One of our major goals for Rational Team Concert 2.0 (RTC) is to support significantly larger development teams on a single server than were supported by our 1.0 release. We knew our Jazz Technology could support more than 250 Developers as soon as we finished our scalability testing and published the Rational Team Concert 1.0 Sizing Guide on developerWorks. So, yes, we were being fairly conservative with RTC 1.0. We were already hosting 800 users on our internal deployment last May (on a single server hosting many Rational Team Concert instances), and all indications were that there was still more capacity available. Since then, our performance team has been scaling up the workloads to simulate larger teams. This week, we’ve made available a special key for the RTC 2.0 Beta 1 that allows you to push it to the limit on the biggest hardware you can find.

Moving well beyond 250 users

We started gradually supporting larger teams with our 1.0.1 release last Fall. We introduced floating licenses, making the limit 250 concurrent users rather than defined users, and we started excluding Contributors from the server limit count. These two changes enabled incrementally larger development teams, and huge numbers of Contributors. Projects working across time zones are able to share floating licenses, possibly multiplying the number of users supported on a server. And the 250-user limit only applies to Developers, so Contributors are no longer counted. The proportion of Contributors varies depending on the project; 25% is probably typical, but an open project like ours at jazz.net may have thousands of community members supported by only 100 developers. Excluding Contributors from the count can sometimes enable a significantly larger number of users to be supported – our Jazz.net deployment of RTC supports over 26,000 Contributors. The reason we’re comfortable doing this is that our monitoring of jazz.net traffic shows that Contributors generate a significantly lower impact on the server than Developers doing the heavy lifting using Source Control.

Aiming significantly higher

For our 2.0 release, we’re aiming significantly higher. While there are details to be worked out, we intend to add a new Enterprise edition for 2.0 that supports much larger teams than RTC 1.0 Standard Edition. Our initial design point is to support a “typical” development project of at least 1000 users on a single Rational Team Concert server instance running on typical hardware. By “typical project” we mean having a composition of 25% Contributors and 75% Developers, which would entail supporting at least 750 concurrent Developers. As far as “typical” hardware, we’re running our 1,000-user workload against roughly the same level of horsepower that supports Rational Team Concert on jazz.net and that was used for the Rational Team Concert 1.0 Sizing Guide: a pair of 2-way/4-core Intel Xeon servers, one for the application server and one for the DB2 database server. Each server has 4GB memory. However, we expect to see even higher numbers on more powerful hardware, taking advantage of some new server platforms in Rational Team Concert 2.0: AIX on IBM P-Series processors, and 64-bit Intel processors running Linux.

Factors affecting number of users

Obviously, there are a lot of variables that can affect the number of users that you might be able to support on a given kind of hardware, or the kind of hardware that you might need to support your “typical” team. We’re going to produce a lot more guidance on this in an updated sizing guide as we get closer to our 2.0 release. Some of the things we know are:

• Builds can generate significant load on the server: a team of 100 continuously building a massive codebase might consume the resources of a much larger development team.
• Database performance is critical, and the communication channel between the app server and database server should be the fastest possible. Co-locating them on the same gigabit network would be a good idea.
• In any secure server, there are lots of calls out to your directory for authentication and user role resolution. Some of our worst performance days had to do with misbehaving LDAP connections.

On the positive side, there are plenty of other options that can increase the number of users that can be hosted per physical server. There are Intel and P-Series servers available now with many cores; these should be capable of hosting multiple large Rational Team Concert servers. This is the architecture for our internal self-hosting environment for development within Rational. We run more than a dozen server instances on one P-Series server.

Test key for 10,000 users

Long story short, we want to update you on our work on scalability for Rational Team Concert, and also to enlist you to help us test the limits. To enable that, we’ve made a key available for the Rational Team Concert 2.0 Beta 1 that removes the 250-user limit on the Standard Edition – just for test purposes until we get the 2.0 product editions into final form. We set the limit to 10,000 users because “One Billion Users! moo-ha-ha-ha-ha” didn’t fit.

The Server Key for Large-Scale Testing can be downloaded from the Rational Team Concert 2.0 Beta 1 “All Downloads” page. Feel free to download it, unlock your Rational Team Concert 2.0 Beta 1, and give it a try. We’d welcome any feedback or discussion. You can also track our progress on testing and documenting RTC 2.0 scalability in this work item: Server performance and scaling data (62550)

Scott Rich
Jazz Foundation PMC Lead