TN0005: Deployment Guide for Jazz Team Server

Summary

This document is intended to address the various issues related to the Jazz Team Server when deploying Rational Team Concert in an organization. It makes recommendations on how to set up and configure Rational Team Concert and the Jazz Team Server. It does not attempt to anticipate every configuration and should be used only as a guide.

Note: The article IBM Rational Team Concert Version 1.0 Server Sizing Guide on ibm.com/developerworks provides additional discussion around deployment planning.

More Information

Hardware

The minimum configuration would be a small team (less than 10 users) running the Express-C edition with Tomcat and a Derby database. They should be able to run on a small departmental PC, a single processor or dual core machine with two gigabytes of memory is sufficient for evaluation and modest use. Four gigabytes of memory may provide better performance as larger amounts of data are created.

An example of a larger configuration is the Jazz Team Server which runs Jazz.net. It is hosted on a pair of IBM two-way Xeon servers, one for the Application Server containing Jazz Team Server application, and one for DB2 and the database. These servers both have four gigabytes of memory to support a medium-sized distributed team. A dual-server configuration is not required to support a large team, but it does allow optimization of the two machines to their task, and generally expands the scalability of the system. Obviously, the two machines should be co-located on the fastest possible network.

The amount of disk space required for a server is also largely dependent on the team size and the volume of artifacts they are working with. As a data point, the Jazz.net repository currently occupies approximately 42 gigabytes of disk space. You can look at the repository metrics reports on Jazz.net or on your own server to see how much space is being used and what kind of artifacts contribute to the space usage. The source control stream size counts may be of interest also.

Web Application Server

IBM WebSphere Application Server and Apache Tomcat are supported for hosting the Jazz Team Server application. The Jazz Team Server installation installs and configures a working Tomcat installation, making it very easy to get started. Tomcat provides a minimal set of management and administration tools, which is most likely to be the limiting factor in a Tomcat installation.

The WebSphere server provides a much richer set of administration, monitoring, and tuning capability than Tomcat. WebSphere can also provide better performance in the area of authentication. The default authentication mechanism is form-based. However, if you choose to switch from form-based to basic-authentication for your Jazz Team Server, you MUST use WebSphere or place an authentication-caching proxy such as IBM HTTP Server or Apache HTTP Server in front of Tomcat. Otherwise Tomcat will authenticate every request and performance of the server will be unusable. If you are using an LDAP directory for authentication, note also that Websphere provides easier administration and greater flexibility for mapping application roles to LDAP groups. Of course, if you already have WebSphere skills and deployed servers, deploying the Jazz Team Server to that environment would be a natural choice. The Jazz Team Server is tested with WebSphere Application Server V6.1 with the latest JVM updates.

With the Rational Team Concert 1.0 release, the Jazz Team Server cannot take advantage of the clustering capability of WebSphere ND. It runs as a single-server application.

Moving from Tomcat to WebSphere is relatively straightforward. There is no difference in the repository data for a Tomcat deployment versus a WebSphere deployment, so the steps are simply to deploy the Jazz Team Server to WebSphere, and configure it to use the existing repository database. There is no migration or massaging of the repository data required. Follow the WebSphere setup instructions in the Server Setup Guide located in (<Jazz Team Server install directory>/jazz/server/install_express.html or <Jazz Team Server install directory>/jazz/server/install_standard.html) to configure Jazz on WAS.

Server Capacity Planning

In the first release, there is no support for moving a Team Concert project (a Project Area and its artifacts) from one server to another. Because of this, it is important to think ahead when assigning projects and users to servers. The user limits per server are strict (Express-C: 10 users; Express: 50 users; Standard: 250 users; for Derby databases the limit is 10 users for all product editions). If the projects deployed on a server grow and the server hits its maximum user limit, all of the projects on that system will be unable to grow. Therefore, we recommend you look at the projects you are deploying, and project their growth such that they can continue to be supported by the server limits. One way to do this, given the imprecision of projecting project growth, is to include a buffer in the server allocations. Leaving a 20% buffer would be a good practice. So, even though a Standard Edition Jazz Team Server can support up to 250 users, planning the growth to only reach 200 would allow for unforeseen growth without hitting the limit.

An organization which is deploying many projects for hundreds of users will need to think carefully and purchase a few more server licenses than the simple math might suggest.

In release 1.0 all users have read access to all data on the server even if they are not formally a member of a specific Project Area. If you must limit access to a project then you should plan to host that Project Area exclusively on it is own Jazz Team Server.

Database

The database platform varies depending on what product edition you have.

The low end Express-C edition includes an embedded Derby relational database and only that database is supported. Derby provides a very simple configuration and essentially zero administration. This is a convenient way to get started.

DB2 Express is provided with the Express and Standard editions of the Jazz Team Server. DB2 provides a much more “industrial-strength” database. It provides a full set of monitoring, tuning, administration tools. It also provides significantly better performance for a larger number of users or large volumes of data. Oracle Database 10g is also supported.

Our experience self-hosting on Jazz and performance testing shows that Derby has a hard time supporting teams larger than 10 users, or repositories larger than about five gigabytes. It’s not really the size of the repository data, but the number of users and the data complexity which becomes the limiting factor. Because of this, we have limited the number of licensed users supported by any server edition running a Derby database to 10 users.

The Express and Standard products include licenses for use of DB2 as the database to store the Jazz repository. A new copy of DB2 can be installed to support the Jazz Team Server, and run with this license. An existing DB2 database server can also be used to support an Express or Standard server. In this case, the server is subject to whatever license terms it was purchased under. The Server Setup wizard in the web UI helps to configure the Jazz Team Server to connect to a new or existing database server.

Moving Your Database to DB2 or Oracle

Moving from one relational database platform to another is relatively straightforward. It uses the same mechanisms as a migration from one Jazz Team Server version to another. The data in the Jazz repository is exported to a TAR file in a database-independent form, and is then imported into the new database in the appropriate form. The repotools command includes a copy command which implements this in a single action. The best practice is to create two configuration files, one for your current configuration and one for the future database configuration. These may be created by hand, by the Jazz Team Server installation program, or by running the Server Setup wizard for a new server install, as would be the case of an upgrade from Express-C to Express. Once you have the two server configurations, you run repotools –copy, and it handles the export from the current configuration to a temporary TAR file, which is then imported to the new database. After the copy, the new configuration can be started, using the newly created repository database.

The downtime required for a repository copy will be dependent on the size of the repository. In the case of the Jazz.net repository, at 42GB, this is currently about a 14 hour operation. Improvements are planned after release 1.0.

Backing up your Database and other Essential Data

BACKUPS, BACKUPS, BACKUPS!!! Before storing the first artifact you care about in your Jazz Team Server, you must have a backup procedure in place. Don’t wait until the first hard drive crash to ask “are we doing backups?” or “how do we know our backups are good?”

For Derby, backups are simple. The Jazz Team Server must be stopped, and then a file copy or ZIP utility can be used to backup the database. Backup the repositoryDB and workItemIndex directories.

For DB2, on-line backups are available and provide a way to keep quality backups with minimal interruption. See the DB2 documentation for details on enabling on-line backups. Some specific settings are required in the area of log file management. Even with a solid backup story, scheduling occasional downtime for a full backup is wise. On Jazz.net, we take on-line database backups nightly and in conjunction with transaction logs, we have capability to restore up to the last committed transaction. Off-line backups occur only when we do migrations or host system configuration changes.

We recommend configuring an Oracle database for ARCHIVE LOG mode. One can use Oracle’s RMAN utility or Enterprise manager to take backups and schedule them on a periodic basis (Daily, weekly, etc.) in conjunction with Archive logs. The same tools will provide functionality to restore the database in case of failure. There might be other components that need to be backed up like instance configuration settings to exactly restore the database to original configuration.

Backup Your Server Configuration Data

In addition to backing up your database you should back up data that defines your server configuration. Some important items are:

Jazz Team Server

1. jazz.war
2. workitemindex diretory
3. If you are backing up RTC 1.0 then the following files should be saved:
   1. teamserver.properties
   2. profile.ini
   3. update-site directory
4. If you are backing up RTC 2.0 then the following files should be saved:
   1. conf directory

Application Server

1. Tomcat: server.xml and web.xml if you change the default values.
2. Tomcat: if you are using Tomcat for user authorization then back up the user registry (tomcat-users.xml).
3. WebSphere: Profiles (WebSphere provides a backup tool for profiles).

Latency and Network Performance

Naturally, the capacity of your wide area network (WAN) and proximity of your users to the Jazz Team Server can have a significant affect on performance and overall throughput. If your Jazz Team Server is used primarily for browsing and managing work items then a low speed WAN is sufficient. Managing large files in work item attachments, source control management, and build will require a higher performing WAN.

The network supporting Jazz development is hosting seven development sites in Europe and North America for approximately 70-80 full time users and thousands of others who are part of the Jazz.net community. The server is in Toronto, Canada. Our most distant development team is in Zurich, Switzerland. They have a two megabit/second network connection which provides satisfactory performance.

As of the Beta 3 build, the Rational Team Concert Eclipse client’s metronome tool allows you to ping your Jazz Team server for upload and download connection values.

User Authentication

Jazz Team Server is secured using HTTPS and either JEE form-based or HTTP Basic authentication. Unsecured connections are not supported. All user authentication is delegated to the application server: Tomcat or WebSphere. The Jazz Team Server supports an LDAP (Lightweight User Directory Protocol) user directory which is the recommended approach. For small teams the Tomcat simple user registry using the tomcat-users.xml file is also supported. In your chosen user directory type (LDAP or Tomcat) users are assigned specific basic repository access permissions that give them read, write, and/or administrative privileges.

References

The following documents contain information relevant to this topic:

• IBM Rational Team Concert Version 1.0 Server Sizing Guide (article on ibm.com/developerworks)
• TN0004: Rational Team Concert 1.0 Editions and Licensing Overview – Tech Tip explaining Rational Team Concert 1.0 editions and licensing options, with more information on how team size affects choice of edition
• The Jazz metronome tool keeps us honest – Blog post by Jazz SCM lead Jean-Michel Lemieux explaining how Jazz Platform developers use the Rational Team Concert Eclipse client’s “metronome” tool to understand and improve network efficiency
• Latest Metrics from Jazz Project Self-hosting Repository – Information on storage used by Rational Team Concert components within the Jazz Project’s self-hosting repository; information provided by Jazz Reports component