Considerations for Rational Team Concert 3.x backups

Ralph Schoon, Jazz Jumpstart Team
Last updated: January 14, 2013
Build basis: Rational Team Concert 3.0 

Once you start using Rational Team Concert it will contain more and more important data. Therefore it is essential to make regular backups of the data to avoid losing it. The Collaborative LifeCycle Management Infocenter contains information on how to backup the RTC 3.x database. To minimize down time, it is possible and recommended to increase the reliability of your deployment using hardware that provides a high reliability and tolerance against failure. However, failure can still happen.  Since whole development teams depend on the infrastructure it is desirable to be able to recover as quickly as possible. This article intends to provide specifics about what to consider when backing up an RTC Server.

The article Backup the Rational solution for Collaborative Lifecycle Management describes the whole process for a CLM deployment including Rational Quality Manager and Requirements Composer.

*Update*

This article was updated to reflect the fact that Defect 245648 prevents from suspending the Fultext indexing service. The current suggestion is to shut down the server to be able to create a valid backup.

Deployment Topology

For the purposes of this article we will assume

• RTC 3.x is installed on one server with both the Jazz Team Server (JTS) capability and the Change and Configuration Management (CCM) capability installed on a single application server
• The installation is a fresh RTC 3.x install without a migration from RTC 2.x

As described in Building products from applications in RTC 3.x the single RTC application has been separated into two product capabilities:

1. Jazz Team Server referred to as JTS in the following document
2. Change and Configuration Management referred to as CCM in the following document

A typical install as assumed above will result in a deployment topology as shown below. The Application Server node has the capabilities JTS and CCM deployed. 

Each capability uses its own database to store data. In the example topology the databases running on the database node are referred to as CCM_DB used by the CCM capability and JTS_DB used by the JTS capability.

Which Data is relevant for Backup Operations?

This is really related to “What could happen?”   In the topology described above there are the following potential failure scenarios you should plan for.

1. Loss of the Database Server node
2. Loss of the Application Server node

The loss of the database server or the application server node would each require a substitute or replacement.  In the worst case this could mean reinstalling or reconfiguring a machine as a replacement.  You can minimize downtime also by providing replacement systems already configured to take over as described in setting up a basic high availability configuration for the application server node. In any case it is necessary to have all required data available.

The data needed to restore a complete RTC 3.x server including application server and data consists of:

1. The databases to store the data
2. The Application Server configuration data 
3. The Jazz Application configuration data created and maintained during setup and administration
4. Any custom server extensions 

In addition to these configuration files, several index files are used by RTC. These index files can either be recreated from the database contents after a restore operation or be included in the backup and restore operation.  The following sections will describe how to locate and backup the data mentioned above.

Practice Backup and Restore

A backup procedure is worthless if data is missing, inconsistent, unreadable or not working for other reasons. You should practice and test your backup and restore procedure on an isolated test infrastructure on a regular basis, to be sure that it works.

Jazz Application Configuration Files

While installing the product you choose an install directory. If you installed using the Installation Manager on Linux for example, the default server install directory is /opt/IBM/JazzTeamServer.  Installing on Windows the server install directory default is C:Program Files(X86)IBMJazzTeamServer. This installation directory will be referred to as {server-install-dir} in the following text.  Please note that using the Windows System default program install directory can cause various issues, especially due to policy and security settings.  It is therefore recommended to use a different install directory on Windows.  The image below shows a typical folder structure after installing on Tomcat.

During set-up and configuration of the Jazz Team Server and its capabilities, several files are changed either automatically or manually. These files contain information used during startup and operation. Some of these files are also updated during administration operations and therefore should to be backed up on a regular basis as well.

Backing up the Jazz application configuration files will be be represented below by the following pseudo code:

backup ${CCM_CAPABILITY_CONFIG}
backup ${JTS_CAPABILITY_CONFIG}

All these files are stored underneath the configuration folder:

{server-install-dir}/server/conf

You have two different options to back up this data, full backup and selective backup.

Full Backup

You should backup the whole folder structure underneath 

{server-install-dir}/server/conf

This option will safely backup all data that might be necessary to restore the Jazz Application Configuration. On the other hand it might back up more data than is actually needed.

Selective Backup

The configuration folder also contains program code which is often not desired to be included in backups. If you want to be more selective during backup, you can back up only the most important files and folders. You can leave out the program data which will be restored when reinstalling the product.

Each capability has its own configuration folder. The Jazz Team Server capability JTS uses the folder 

{server-install-dir}/server/conf/jts

The CCM capability stores its data in the folder 

{server-install-dir}/server/conf/ccm  

Each capability has its own configuration files as shown below. 

For each capability JTS and CCM you should at least back up the following files

teamserver.properties
log4j.properties

Back up Custom Server Extensions

In case you have deployed custom server extensions you should backup the deployed binaries.  Make sure the deployed zip/jar file is included in your backup.  If you do a full backup all of the other configuration data should be backed up already.

In case you do a more selective backup and have deployed custom server extensions, you should backup the deployed binaries and in addition back up the respective provisioning .ini file or all of the provisioning .ini files in the folder:

provision_profiles/

Backup Your Application Server Configuration

You should back up your application server configuration data, in order to be able to restore it later.  How to backup this data depends on the Application Server being used. 

Backing up the application server configuration files will be be represented below by the following pseudo code:

backup ${APPSERVER_CONFIG} 

RTC on Tomcat

When installing RTC3.x on Tomcat the Application Server Configuration data resides in {server-install-dir}/server/tomcat/conf/. You should back up the all files that were manually modified during the setup process.  The following following files are typically modified during setup and should thus be backed up:

{server-install-dir}/server/tomcat/conf/server.xml
{server-install-dir}/server/tomcat/conf/web.xml

If you provide a server certificate you should include this into your backup as well.  In case you are not sure, information about the storage location of the used certificate can be found in the server.xml file. The element Connector contains a property keystoreFile that contains the file location.

In case you edited the capabilities web.xml files, for instance when using Tomcat with LDAP,  you also want to include these files into your backup. The web.xml files to backup are present at:

{server-install-dir}/server/tomcat/webapps/ccm/WEB-INF/web.xml
{server-install-dir}/server/tomcat/webapps/jts/WEB-INF/web.xml

If you are using Tomcat for user authorization you should also back up the user registry file

{server-install-dir}/server/tomcat/conf/tomcat-users.xml

RTC on WebSphere

When installing RTC3.x on WebSphere you should backup the profile used to deploy RTC. See the WebSphere Application Server Infocenter for your version of the WebSphere application server for more information.

Back up your Databases

In RTC3.x the JTS and CCM capabilities use their own database to store the data. You will need to backup both databases. 

Since there are two databases to back up and restore and the applications interact, the order the sequence of the backup and restore operations is important. 

Data of the JTS capability is automatically transferred into the CCM capability. This implies that any changes in the JTS repository that are missing in the CCM repository are automatically applied at a later time. To avoid conflicts and ensure consistency the CCM database must be backed up before the JTS database. 

When restoring from an online backup make sure to restore the CCM database from a point in time shortly before the JTS database backup.

Backing up the databases will therefore be be represented in the order below by the following pseudo code:

dbbackup ${CCM_DB}
dbbackup ${JTS_DB}

Derby Database

In the case where a Derby database is used, the application server server must be shut down to perform the database backup.

To back up the databases, back up the complete directories used by Derby as repository.  By default the directory is called  repositoryDB. The default location for this directory is the derby folder in the configuration folder of each capability:

{server-install-dir}/server/conf/ccm/derby/repositoryDB
{server-install-dir}/server/conf/jts/derby/repositoryDB

Please verify the location of your setup by checking the property com.ibm.team.repository.db.jdbc.location in the teamserver.properties file of each capability.

Other Databases

Please see the the Release Notes for the supported database management systems. Please also see the Infocenter and the Library for suggestions and links on how to backup other databases. For detailed information on how to backup your database, please refer to the documentation for the database management system you are using.

In general the options provided by enterprise database management systems are:

Off line Backup

It is always possible to do an off line backup. To do so you must stop the application server before you perform the database backup. The shutdown ensures that no user interaction occurs during the backup.  You should also backup the index files while the server is shut down. 

On line Backup

Some database management systems such as DB2 support on line backup. This can be used to continuously backup and later restore the databases.  See this TIP for more details for DB2.

Defect 245648 prevents from continuously running while backup. See Index files, especially Backing Up the Fulltext Index Files for details. You can run an online backup for the database but it is necessary to stop the server for backing up the index files. 

If using on line backup you still want to backup the Jazz application configuration files as well as the application server configuration files. You will most likely not be able to backup and restore the configuration files consistently with the on line backup. However, these files are only changed when configuring the application server or administrating the capabilities of the server configuration. It is suggested that you backup these files after performing any of these tasks. Otherwise you need to redo changes that didn’t make it into the last backup.

You should consider trying to backup the index files consistent with the on line database backup. For the JFS index, parts that can be backed up on online, have a backup from a time before your online backup. 

You have to shut down the server to backup the Fulltext index. This would be a good opportunity to create a consistent offline back up too. If this is not possible it will be necessary to rebuild the index files after restoring the databases.

Index Files

In addition to the data mentioned above, JTS and CCM in RTC 3.x use index files for various query and search operations that need back up. It is possible to recreate the index files from a restored database backup. Recreating the index files is slower than restoring them from a index files back up. To speed up a restore operation, the index files can be backed up with the databases.

To prevent corrupting the index data while performing the back up you should shutdown the server. 

Backing up the indexes will be be represented below by the following pseudo code:

backup ${JTS_INDEXES}
backup ${CCM_INDEXES}

Backing Up the JFS Index Files

The location of the JFS index files can be found in the teamserver.properties file for each capability as the values of the following properties:

com.ibm.team.jfs.lucene.directory
com.ibm.team.jfs.lucene.history.directory
com.ibm.team.jfs.jena.tdb.history.dir
com.ibm.team.jfs.jena.tdb.storage.dir

Backing Up the Fulltext Index Files

The location of the Fulltext index files can be found in the teamserver.properties file for each capability as the values of the following properties:

com.ibm.team.fulltext.indexLocation

Backup Pseudo Code

A complete off line backup procedure would perform the following activities:

shutdown ${SERVER}
backup ${APPSERVER_CONFIG}
backup ${CCM_CAPABILITY_CONFIG}
backup ${JTS_CAPABILITY_CONFIG}
backup ${CCM_INDEXES}
backup ${JTS_INDEXES}
dbbackup ${CCM_DB}
dbbackup ${JTS_DB}
startup ${SERVER}

Restore Pseudo Code

A complete restore procedure will perform the following activities:

restore ${APPSERVER_CONFIG}
restore ${CCM_CAPABILITY_CONFIG}
restore ${JTS_CAPABILITY_CONFIG}
restore ${CCM_INDEXES}
restore ${JTS_INDEXES}
dbrestore ${CCM_DB} 
dbrestore ${JTS_DB}

Please note as stated above, when restoring from an online backup make sure to restore the CCM database from a point in time shortly before the JTS database backup.

RTC3.x after Updating from RTC2.x

In the case of an installation that has been upgraded from RTC 2.x to RTC 3.x the CCM capability is installed using the path /jazz instead of /ccm. Therefore all occurrences of /ccm in this article above must be replaced with /jazz.

More complex scenarios

With RTC 3.x you have the option to install the JTS and the CCM capability on different servers. In that case you must perform all backup steps for each individual server. 

Appendix

Stop JFS Indexing Services

You can stop and resume JFS indexing either by shutting down the server or using the repotools. It would be possible to stop indexing and backing up the indexes while the server is running. This scenario is not presented in the article because Defect 245648 prevents this from working for the Fulltext index, which makes it necessary to take the server offline. 

It is only possible to stop the JFS indexer that creates the indexes for the following properties. 

com.ibm.team.jfs.lucene.directory
com.ibm.team.jfs.lucene.history.directory
com.ibm.team.jfs.jena.tdb.history.dir
com.ibm.team.jfs.jena.tdb.storage.dir

To stop the indexing services for JTS and CCM use:

repotools-ccm -suspendIndexer ${CCM_SERVER} 
repotools-jts -suspendIndexer ${JTS_SERVER} 

To resume the indexing services use:

repotools-ccm -resumeIndexer ${CCM_SERVER} 
repotools-jts -resumeIndexer ${JTS_SERVER} 

  In the above example, ${CCM_SERVER} and ${JTS_SERVER} represent the repository URL of your Server in the format

repositoryURL=https://<yourCCMServer>:<YourCCMServerPort>/ccm
repositoryURL=https://<yourJTSServer>:<YourJTSServerPort>/jts

Recreating the Index Files

It is possible to recreate the index files from a restored database backup. Please be aware that recreation of the index files is an expensive operation and needs to be performed with the server shut down.

Recreate the Fulltext Index

You can recreate the Fulltext index files using the command:

repotools-${APP} -rebuildTextIndices

Recreate the JFS Index

You can recreate the JFS index files using the command:

repotools-${APP} -reindex all

Summary

This article described the data and steps required to back up an RTC3.x installation. It provides several options on how to perform a backup and also provides information to perform backups for more complex Jazz deployments. 

---

For more information

• Jazz Foundation Server Administration
• Jazz Administration Plan
• Jazz Deployment Guide

---

About the author

Ralph Schoon is a member of the Jazz Jumpstart team.  The Jazz Jumpstart team is a worldwide group of development specialists who bring new and advanced Jazz-based technologies to customers. Please direct feedback and comments to ralph.schoon@de.ibm.com .

Copyright � 2010 IBM Corporation