Tuning the Rational Team Concert 4.0 server
Introduction
This article serves as a source for optimization settings and troubleshooting strategies in order to ensure that you get the best performance out of your Rational Team Concert (RTC) deployment.
This guide does not replace the RTC installation instructions. To understand this document, users should have already had some experience getting the Jazz Team RTC Server installed and working with the appropriate web container and database. This guide also complements the RTC server sizing guide as a source for deployment recommendations.
The Jazz Team RTC Server is only part of the CLM server installation and although this guide does not make direct recommendations for the other pieces (JTS, etc..) it should be noted that the advise given here generally applies also to the other pieces as well. The connection characteristics such as latency and bandwidth given for the dual tier configuration of the Jazz Team RTC server apply equally to the connections between the Jazz Team RTC server and the JTS server as well.
The Jazz Team RTC Server consists of two distinct logical parts, each of which can be tuned separately, but also may require tuning relative to each other, depending upon the conditions of the deployment. As illustrated in figure 1, the Jazz Team RTC Server consists of two logical tiers which can be deployed either on a single piece of hardware, or spread across two separate server machines.
Figure 1 : Jazz Team RTC Server logical structure
How is my server currently running?
There is a web page which describes the server runtime statistics for the duration of the server uptime.
<your jazz server uri>/service/com.ibm.team.repository.service.internal.counters.ICounterContentService/
This page provides a table which contains content similar to that in figure 2.
Figure 2 : Sample server statistics
Source Control operations are often the most resource intensive, so you can use the average times for IScmService as an indicator of the relative health of your server. Operations like createBaseline() and createChangeSet() should average out to be less than a second of runtime. Methods like createWorkspace() and acceptCombined() are amongst the most resource intensive server operations, and average approximately 4 seconds on jazz.net. For a description of the jazz.net deployment topology, it is described in this presentation in the Jazz.net library.
In most environments, the most commonly called method is IVersionedContentService#GET() which is what is used to download the content for Source Control loads and updates. If you are seeing service counters significantly higher than what we have described here, then hopefully these tips and suggestions can improve your performance.
Deployment basics
The Jazz Team RTC Server performs better when it runs on machines that are not shared with other applications. This is true for the database as well as the application server. Running other applications on the hardware takes away from the amount of memory that the database and application can use to optimize their performance and caching. This also can include firewalls and virus scanners. Unnecessary services and processes should be disabled.
The Jazz Team RTC Server cannot be guaranteed enterprise performance characteristics when run virtualized. We have not tested the capacity of the RTC Server in a virtual environment.
In our testing, we have found that best performance was found on 64-bit machines, running 64-bit operating systems and the full 64-bit Jazz Team RTC Server stack, including a 64-bit database.
Tuning the server machine(s)
Storage
Storage performance is by far the most critical element to optimize for the Jazz Team RTC Server. For a true enterprise configuration and expected workload, we recommend that the server has many fast disks, combined in a RAID configuration, for optimal performance and reliability. RAID-10 is recommended for files that need redundancy like the database tablespace. RAID-0 can be used for disk locations which do not require the redundancy, like temporary tablespaces and temp directories.
For maximum performance, you can partition each hard drive you have into 2 parts, then include the 1st part of each hard drive in your RAID-10 device and the second part into the RAID-0. If your Hardware RAID doesn’t support splitting up partitions like this, then you can use Software RAID to emulate the benefits of Hardware RAID. Also, for the disks, you should ensure that all disk caches and DMA transfer modes are enabled.
If you are planning on using network storage, then you can improve your performance by ensuring that your network link is of extremely low latency : using Ethernet for such a network link is less than optimal.
Operating System Settings
Ensure that the swap partition/file is also placed on the fast storage.
Windows
Change processor scheduling to prefer background services in Control Panel > System > Advanced > Performance Settings > Advanced.
Adjust memory allocation for best performance of programs in Control Panel > System > Advanced > Performance Settings > Advanced.
Choose Maximize data throughput for network applications in Control Panel > Network Connections > <Connection Name> > Properties > File and Printer Sharing for Microsoft Networks > Properties for each listed <Connection Name>
32-bit Windows
32-bit Windows is not a recommended platform for Enterprise-class performance or scalability. With that in mind, if you are stuck on 32-bit Windows, there are a couple of options available for you to try to improve the performance of the Jazz Team RTC Server.
If you have more than 3GB of memory, you can enable applications to access up to 3GB (instead of just 2GB) by adding /3GB to boot.ini using these instructions from Microsoft. Be sure to test this first as Windows can become very unresponsive if any of your drivers do not support the /3GB switch. If this happens you will have to boot to safe mode to remove the /3GB switch.
If you have more than 4GB of memory, add /PAE to boot.ini to make use of it.
You should also give the user running the DB permission to Lock Pages in Memory.
Linux
Different database vendors recommend different settings. We will link or describe the settings in the appropriate sections below in tuning the database.
Tuning the database
General tips
DB2
We recommend using DB2 9.7 instead of DB2 9.5 or DB2 9.1, although 9.1 and 9.5 are supported. DB2 9.7 has many performance fixes and better automatic tuning features that you can take advantage of.
In addition to the basic setup instructions, we find that some people miss an additional optimization that we recommend in the setup documentation.
On 32-bit operating system, ensure that INSTANCE_MEMORY is set below 3GB (2GB on Windows without /3GB). Do not rely on INSTANCE_MEMORY set to automatic to stay below this upper limit.
Starting with 3.0 the RTC server supports DB2 running in currently committed mode. This mode increases concurrency and reduces deadlocks. To enable this mode turn on the cur_commit parameter.
Using a RAID Tablespace
If you are using RAID for your tablespace storage, you should not be letting repotools create the tablespace, as there are options that you can specify to let DB2 know that it is running on RAID and that it can optimize itself for having a explicitly fast tablespace.
For a new installation, you should instead create a LARGE tablespace called “CONTENTTS” by following the optimizing DB2 For RAID guide. Make sure to explicitly set EXTENTSIZE, OVERHEAD, TRANSFERRATE and DB2_PARALLEL_IO. Note that the OVERHEAD and TRANSFERRATE parameters apply even if you are not using RAID, so be sure to set them.
If you already have a CONTENTTS tablespace built by repotools, but you want to convert to a new tablespace, then you would need to follow the following steps. This will require some downtime for the Jazz Server, and we recommend that you take a backup of your database prior to performing these steps. If you followed the standard setup instructions, your database should be named JAZZ so you can replace <dbname> with JAZZ throughout this process.
- Shutdown the Jazz Team RTC Server
- Navigate to a new working directory (that ideally is on a fast disk with lots of free space)
- run
db2look -e -d <dbname> > jazzschema.ddl - run
db2move <dbname> export
- run
db2 drop db <dbname> - Create the new database. On windows:
db2 create database <dbname> on <raid drive> using codeset UTF-8 territory en PAGESIZE 32768. On UNIX:db2 create database <dbname>using codeset UTF-8 territory en PAGESIZE 32768.
- create the tablespace/logs in the right place as per the above instructions
- run
db2 -tf jazzschema.ddl - run
db2move <dbname> load - restart the Jazz Team RTC Server
The same process can be applied to put the USER TEMPORARY tablespace on the RAID.
On Linux, there are instructions for changing your kernel parameters. One additional area of note would be to ensure that the response to the UNIX command “ulimit -n" >= MAXFILOP. MAXFILOP is a property that can be found by running the command db2 get db cfg when connected to the Jazz database.
Troubleshooting DB2 performance
All of these commands can be run while the Jazz Team RTC Server is running.
Run db2 get health snapshot for all on <dbname> to see if the DB2 health monitor is detecting any problems. Skilled database administrators can also run db2diag -l Warning,Error,Severe to see if there are other known issues from DB2’s perspective.
If things are running particularly slow, you can run “db2top” during these operations and press “B” (Bottlenecks) to see where DB2 is spending time. This will give you an idea of what area can be tuned, and as well can provide useful data to contact support with.
Run db2mtrk -d and make sure one of the bph (<#>) values has a reasonable amount of memory (256MB is minimum recommended for RTC). If the value is low, and you are using a 32-bit OS then INSTANCE_MEMORY can be set too high (most likely) and memory allocation failed.
If the auto-tuning feature does not set things as you would expect, then you can adjust the memory property manually for the buffer pool with db2 alter bufferpool <bpname> size <npages>. If you wish for DB2 to use 2 GB of RAM, then perhaps you would allocate 1GB for the bufferpool. 1GB of memory divided by 8 K page size = 131072 pages. If you want to continue manually tuning things, then you would also want to increase or update the value of the value of LOGBUFSZ and DBHEAP as shown when running db2 get db cfg.
Oracle
You can disable the Oracle recycle bin by adding “recyclebin=off” to init<SID>.ora (where <SID> is the oracle system identifier); or alternatively run alter system set recyclebin=off scope=both; in the SqlPlus command prompt.
To disable statistics history retention: run exec DBMS_STATS.ALTER_STATS_HISTORY_RETENTION(0); in the SqlPlus command prompt.
To enable asynchronous commit to speed up database writes, please add “COMMIT_WRITE = BATCH, NOWAIT” to init<SID>.ora or alternatively you can run alter system set commit_write = batch,nowait scope=both; in the SqlPlus prompt.
Ensure that SGA_TARGET (shared memory) is less than the size of your physical memory. This value can be checked at by running show parameter sga_target at the SqlPlus prompt. To set it to 700MB, you would run alter system set sga_target = 700M scope=both; at the SqlPlus prompt. On 32-bit operating system it should be less than 3 GB as well (2 GB on Windows).
For more information about tuning Oracle performance, refer to the Oracle performance tuning guide, parts I – III.
Additionally there are operating system specific tuning guides for Oracle. Refer to your system specific tuning guide for Oracle, for example: Windows IA-32 (and VLM tuning for Windows IA-32 if you have more than 4GB of memory) or Windows AMD64
SQLServer
Follow the Optimizing server performance instructions. On 32-bit Windows, if you have more than 4GB of memory, you can enable the AWE option by following the Managing Memory for Large Databases instructions (including all subsections).
Starting with 3.0 the RTC server supports SQLServer running in row versioning mode. This mode increases concurrency and reduces deadlocks. To enable this mode turn on the READ_COMMITTED_SNAPSHOT option.
Tuning the application server
For all the databases we currently support, the database automatically manages the number of concurrent connections or clients that can be served. However, it is helpful to explicitly decide how many concurrent client requests you want to allow to process at a time. Lets give this value a name (max_clients ). Note that max_clients is not simply the number of users of the server. It is in fact much smaller than that because not all users are executing a request at the same time. If this is a new server that doesn’t have historical data about what the maximum number of concurrent requests is, a good starting estimate is 10% of the number of clients expected to be using the server at any one time.
A well configured, “busy” server does not usaully need to service more than 10 specific clients at an exact point in time, but you should play around with this number if you encounter any “Service Temporarily Unavailable” type connection errors. Increasing this number will increase parallelism on the Jazz RTC Team Server, but it will also use more memory and file descriptors. On UNIX, you should ensure that the ulimit -n command is greater than or equal to max_clients * 100 + 512 (this setting is normally persisted in /etc/security/limits.conf).
TeamServer Configuration Parameters
You can update different TeamServer configuration options either by altering the settings in the admin web UI (through the Advanced Properties link), or by updating the appropriate property name and value in the teamserver.properties file located in jazz/server/conf/jazz. The following table describes how to update the appropriate settings.
Admin Web UI – Advanced Property | teamserver.properties property name | Value to set to |
|---|---|---|
| Core Repository Component – Content Temporary Directory | com.ibm.team.repository.tmpdir | <path should be on fast storage> |
| Core Repository Component – JDBC Connection Pool Size | com.ibm.team.repository.db.jdbc.pool.size | max_clients * 25 |
| Core Repository Component – Mediator Pool Size | com.ibm.team.repository.rdb.mediator.pool.size | max_clients * 25 |
| Jazz Foundation Services – Index Directory | com.ibm.team.jfs.index.root.directory | <path should be on fast storage> |
| SCM – Versionable Content Temporary Directory | com.ibm.team.scm.vcs.tmpdir | <path should be on fast storage> |
| SCM – Content Temporary Directory | com.ibm.team.scm.tmpdir | <path should be on fast storage> |
| FullText – Index Location | com.ibm.team.fulltext.indexLocation | <path should be on fast storage> |
The rationale behind the multiplier 25 is as follows: loading or updating an eclipse workspace will create a burst of parallel content fetch requests, up to 25 threads at once, depending upon how the client is configured. In preparing for the worst case, your server should be prepared for such a burst in Jazz traffic.
Tomcat Configuration
Edit server/tomcat/conf/server.xml. You will see a section like the following:
<Connector port="9443" connectionTimeout="20000" maxHttpHeaderSize="8192" maxThreads="150" minSpareThreads="25" maxSpareThreads="75" enableLookups="false" disableUploadTimeout="true" acceptCount="100" scheme="https" secure="true" clientAuth="false" keystoreFile="ibm-team-ssl.keystore" keystorePass="ibm-team" sslProtocol="SSL_TLS" algorithm="IbmX509" URIEncoding="UTF-8" />
Set maxThreads >= (max_clients * 50) + minSpareThreads for enabled Connectors. Tomcat will try and do its own thread pooling, and you need to set the maximum number of threads to a fairly high threshold in order to ensure that tomcat will allow max_clients * 10 threads to be used. By default, tomcat will not use all the threads that you have specified here.
To increase the amount of memory allocated to the tomcat server, edit server.startup, change -Xmx parameter to a higher value. We will recommend some setting in the next section.
Tuning the overall deployment
Single Tier Deployment
In a single tier deployment, your application storage can be the same as your database storage. For a 64-bit machine with enough processing power and RAM, this can be your best bet for the highest performance and capacity. The following table lists off some example distributions of memory allocation between the Database and Application servers.
Architecture | Available RAM | Database | Application |
|---|---|---|---|
| 32-bit Windows | 4GB | 2GB | 1GB |
| 64-bit Windows | 8GB | 4.5GB | 3GB |
| 32-bit Linux | 4GB | 2.5GB | 1.5GB |
| 64-bit Linux | 8GB | 4GB | 3.5 GB |
| 64-bit Linux | 16GB | 11GB | 4.5GB |
| 64-bit Linux | 32GB | 20GB | 10GB |
Dual Tier Deployment
The socket which connects the two physical tiers (application and database servers) must be as low latency (less than 0.3 ms) and high speed (1 GB/s encouraged) as possible. The two tiers should be on the same subnet, without a firewall between them.
In a dual-tier deployment, we would recommend giving each server process as much memory as is available.
Conclusions
The Jazz Team RTC Server can be optimized for you deployment in a wide variety of ways. This tuning guide is the sum of our experience in testing, debugging and troubleshooting different Jazz Team RTC Server configurations over the past 4 years. As we discover more capabilities to tune, we will update this document accordingly.
Copyright © 2012 IBM Corporation