Backup the Rational solution for Collaborative Lifecycle Management

Ralph Schoon, Jazz Jumpstart Team
Last updated: January 14, 2013
Build basis: Rational solution for Collaborative Lifecycle Management 2011 – Rational Team Concert, Rational Quality Manager, Rational Requirements Composer 3.0.1.1. This article is also valid for the product versions 4.x up to 4.0.1.

Once you start using the CLM Solution it will contain more and more important data. Therefore it is essential to make regular backups of the data to avoid losing it. The Collaborative Lifecycle Management Infocenter contains information on how to backup the databases. To minimize down time, it is possible and recommended to increase the reliability of CLM Solution by using hardware that provides a high reliability. However, failure can still happen. Since whole development teams depend on the infrastructure it is desirable to be able to recover it as quickly as possible. This article intends to provide specifics about what to consider when backing up your CLM Solution.

*Update*

This article was updated to reflect the fact that Defect 245648 prevents from suspending Fulltext indexing. The current suggestion is to shut down the server to be able to create a valid backup, including the Fulltext index.

Deployment Topology

For the purposes of this article we will assume the CLM Solution is deployed in a fully distributed topology.

• The Jazz Team Server (JTS) is installed on its own server e.g. JTS_SERVER
• The Change and Configuration Management application (RTC) is installed on its own server CCM_SERVER
• The Quality Management application (RQM) is installed in its own server QM_SERVER
• The Requirement Management application (RRC) is installed in its own server RM_SERVER
• For simplicity we assume a new install without a migration from prior versions and the context roots are /jts, /ccm, /rm, /qm
• The databases are hosted on one or more servers

As described in Building products from applications the CLM Solution is now build around a central Jazz Team Server and several applications that share this Jazz Team Server:

1. The Jazz Team Server is referred to as JTS in the following document
2. The Change and Configuration Management application is referred to as CCM in the following document
3. The Quality Management application is referred to as QM in the following document
4. The Requirement Management application is referred to as RM in the following document

The assumptions above will result in a deployment topology as shown below. Each application node has an Application Server installed and the application is deployed on it.The applications JTS, CCM and QM have their own own database to store data. RM uses the JTS to store its data. In the example topology the databases hosted on the database nodes are referred to as JTS_DB used by JTS and RM, CCM_DB used by CCM and QM_DB used by QM.

In case a Data warehouse such as Rational Reporting for Developer Intelligence or Rational Insight is set up, keep in mind to set up a separate backup procedure for its database and configuration data is also needed.

Which Data is Relevant for Backup Operations?

This is really related to “What could happen?”. In the topology described above there are the following potential failure scenarios you should plan for.

1. Loss of one or more Database Server nodes
2. Loss of one or more Application Server nodes

The loss of the database server or the application server node would each require a substitute or replacement. In the worst case this could mean reinstalling or reconfiguring a machine as a replacement. You can minimize downtime also by providing replacement systems already configured to take over as described in setting up a basic high availability configuration for the application server node. In any case it is necessary to have all required data available.

The data needed to restore a complete deployment as described above consists of:

1. The databases with consistent data
2. The Application configuration data created and maintained during setup and administration
3. The Application Server configuration data for each Application Server
4. The JFS Index Files used by each application
5. The Fulltext Index files for applications CCM and QM.
6. Any custom server extensions and applied Patches

Practice Backup and Restore

A backup procedure is worthless if data is missing, inconsistent, unreadable or not working for other reasons. You should practice and test your backup and restore procedure on an isolated test infrastructure on a regular basis, to be sure that it works.

Synchronize Server Clocks

For normal operation as well as for backup and restore it is important, that the clocks of your machines that run the CLM Solution are closely synchronized. The applications that require the synchronization provide setting up a connection to a NTP server to monitor the machine clocks in the advanced server properties.

Backup the Jazz Application Configuration Files

While installing the products you choose install directories. If you installed using the Installation Manager on Linux for example, the default server install directory is /opt/IBM/JazzTeamServer. Installing on Windows the server install directory default is C:Program Files(X86)IBMJazzTeamServer. This installation directory will be referred to as {server-install-dir} in the following text. The image below shows a typical folder structure after installing all applications on Tomcat.

During set-up and configuration of the Jazz Team Server and the other applications, several files are changed either automatically or manually. Some files that are updated during administration operations and should therefore be included in a regular backup as well.

In this article the backup of the Jazz Application Configuration Files will be represented by the following pseudo code:

backup ${APP}_CONFIG

In the pseudo code ${APP} represents the application node that needs to be backed up. In the pseudo code for a specific application ${APP} needs to be replaced with the acronym (JTS, CCM, QM, RM) representing the application. This pattern is kept in the article. The pseudo code below would do the backup for all applications.

backup CCM_CONFIG
backup QM_CONFIG
backup RM_CONFIG
backup JTS_CONFIG

Files to Include in the Jazz Application Configuration Files Back Up

All the Jazz Application Configuration Files are stored in one or more folders underneath the configuration data folder:

{server-install-dir}/server/conf

The options for backing up the Jazz Application Configuration Files are a full backup and selective backup.

Full Backup

You can backup the whole folder structure underneath

{server-install-dir}/server/conf

for each application. This option will safely backup all data that might be necessary to restore the Jazz Application Configuration. On the other hand it backs up more data than is actually needed.

Please note: By default the index files for the JFS and the Fulltext index are also stored in this location. If you use a full backup for the folder you need to make sure the backup of the index files is successful or change the server configuration and place the index files into a location outside of the {server-install-dir}/server/conf folder. See below for more information.

Selective Backup

The configuration folder also contains program code which is often not desired to be included in backups. To be more selective, it is possible to only include the most important files and folders into the back up. This can especially exclude all data that is installed with the product.

Each application has its own configuration folder. Please note that an application can use more than one folder. The folders need to be be backed up together.

The Jazz Team Server JTS application uses the following folders for JTS and the Lifecycle Project application (LPA):  

{server-install-dir}/server/conf/jts
{server-install-dir}/server/conf/admin

The CCM application stores its data in the folder

{server-install-dir}/server/conf/ccm

The QM application stores its data in the folder

{server-install-dir}/server/conf/qm

The RM application stores its data in the folder

{server-install-dir}/server/conf/rm

Each application has its own configuration files as shown below.

For each application you should at least back up the following files from the configuration folder:

teamserver.properties 
log4j.properties

In general you should back up all files of the form:

*.properties

Some applications store data as RDF. Check for files of the following form and back them up in case there are any:

*.rdf

Other files created during the install are typically not modified. Check the modification date and time if in doubt and include files that changed after the install finished.

Configuration Folder

As mentioned above, if you use this as a general advice for Jazz based solutions e.g. to backup Rational Design Manager, or for newer versions of the tool, make sure you back up all the relevant data in all available configuration folders in 

{server-install-dir}/server/conf

Jike JTS, Design Manager uses more than one configuration folder, in this particular case dm and vvc. You  need to make sure to catch all relevant data from these folders.

Don’t add any folders to the {server-install-dir}/server/conf folder. Especially don’t create copies of the existing folders and store them in this folder. This will cause issues with your application for example if you run setup again e.g. in an upgrade scenario.

Back up Custom Server Extensions

A full backup contains all the required data. If you have deployed any custom server extensions, and do a selective backup, it is advisable to either back up the corresponding files or at least have the files available for rapid redeployment. The files needed to deploy custom server extensions are typically stored in the folder

{server-install-dir}/server/conf/${APP}/provision_profiles

The files that need to be backed up are the provision profile .ini file and the corresponding update site folder.

Back up the deployed patches

A full backup contains all the required data. If you have deployed any patches, and do a selective backup, it is advisable to either back up the corresponding files or at least have the files available for rapid redeployment. See the patch description for which files needed to be included in a backup.

Backup the Application Server Configuration

You should back up the application server configuration data for all application servers, in order to be able to restore it later. How to backup this data and which data needs to be included in the back up depends on the application server in use. The application server configuration changes only due to administrative tasks. It should be sufficient to use an incremental back up once a day.

Backing up the application server configuration files for the application ${APP} will be be represented below by the following pseudo code:

backup ${APP}_APPSERVER_CONFIG

The pseudocode below shows the steps needed to back up all application server configuration files for all nodes.

backup CCM_APPSERVER_CONFIG
backup QM_APPSERVER_CONFIG
backup RM_APPSERVER_CONFIG
backup JTS_APPSERVER_CONFIG

Tomcat

When installing on Tomcat the application server configuration data resides in {server-install-dir}/server/tomcat/conf/. Some of these files are modified during the setup process and during operation and should be included in the backup procedure.

The following files are typically modified during the setup and should be backed up:

{server-install-dir}/server/tomcat/conf/server.xml
{server-install-dir}/server/tomcat/conf/web.xml

If you provide a server certificate, you should include this into your backup as well. In case you are not sure, information about the storage location of the used certificate can be found in the server.xml file. The element Connector contains a property keystoreFile that contains the file location.

In case you edited the applications web.xml files, for instance when using Tomcat with LDAP, you also want to include these files into your backup. The web.xml files to backup are present at:

{server-install-dir}/server/tomcat/webapps/${APP}/WEB-INF/web.xml

For example:

{server-install-dir}/server/tomcat/webapps/jts/WEB-INF/web.xml
{server-install-dir}/server/tomcat/webapps/ccm/WEB-INF/web.xml

If you are using Tomcat for user authorization you should also back up the user registry file

{server-install-dir}/server/tomcat/conf/tomcat-users.xml

WebSphere

When using WebSphere, backup the profile used to deploy the application. See the WebSphere Application Server Infocenter for your version of the WebSphere application server for more information. One option is to use the command manageprofiles.

Back Up the Databases

It is necessary to backup all databases used by the applications. The RM application uses the JTS to store its data and therefore does not have a separate database to backup. In case you use a data warehouse you should also set up a backup procedure for its database.

Backing up the application database for application ${APP} will be be represented below by the following pseudocode:

backup ${APP}_DB

The pseudocode below represents all steps and the sequence needed to backup for all databases for all applications:

backup CCM_DB
backup QM_DB
backup JTS_DB

Ensure a Consistent Database Backup

When backing up a database, it is important to make sure the database backup is complete and consistent from a transaction perspective. A backup is consistent if taken offline, while the database is inactive, or when it is locked for write operations. Some enterprise databases allow consistent online backups with transaction logging or vendor specific procedures and tools.

There are several databases to back up and the applications interact. JTS creates elements in the registered applications. The JTS could not create an element if it is already there, because the database of the registered application is ahead of the JTS. To avoid conflicts and ensure consistency, the databases for all registered applications must be backed up to a point in time equal or earlier than the JTS database. In the example topology, this holds for the CCM as well as the QM database. This has an impact on the database restore operation especially for online backups.

Derby Database

In case a Derby database is used, the application server server must be shut down to perform the database backup.

To back up the databases, zip and back up the directory and its content used by Derby as repository. By default the directory is called repositoryDB. The default location for this directory is the derby folder in the configuration folder of each application:

{server-install-dir}/server/conf/ccm/derby/repositoryDB
{server-install-dir}/server/conf/qm/derby/repositoryDB
{server-install-dir}/server/conf/jts/derby/repositoryDB

Please verify the location of your setup by checking the property com.ibm.team.repository.db.jdbc.location in the teamserver.properties file of each application.

Enterprise Databases

Please see the system requirements on the ‘Getting Started’ tab of the individual product download page for the supported database management systems. Please also see the Infocenter and the Library for suggestions and links on how to backup other databases. For detailed information on how to backup your database, please refer to the documentation for the database management system you are using.

In general the options provided by enterprise database management systems are:

Offline Backup

It is always possible to do an offline backup. Stop the application server before performing the database backup. The shutdown ensures that no user interaction occurs during the backup. This ensures the backup is consistent. You can also consistently backup all other data related to the server while it is shut down.

Online Backup

Some database management systems such as DB2 support online backup. This can be used to continuously backup the databases. For more details about DB2 online backup see this article. 

If using online backup you still want to backup the Jazz application configuration files as well as the application server configuration files. You will most likely not be able to backup and restore the configuration files consistently with the online backup. However, these files are only changed when configuring the application server or administrating the capabilities of the server configuration. It is suggested that you backup these files after performing any of these tasks. Otherwise you need to redo changes that didn’t make it into the last backup. 

You also need to have a consistent backup of the JFS Index Files and the Fulltext index files as described below. Due to Defect 245648 you need to back up the Fulltext index files while the server is shut down. This would be a good opportunity to do an consistent offline backup of the databases. Since the Fulltext index can not be backed up independent of the database, it would be necessary to recreate the Fulltext index if the database is not backed up consistent to it.

Backup the JFS Index Files

The applications use JFS index files for various query and search operations that need to be backed up. The JFS index files can be backed up independent from the database backup.

Ensure a Consistent Backup of Database and Index Files

In order to create a consistent backup of the JFS index files and restore from it, you should backup the JFS index files some time before or during  the backup of the database. The indexing service will recreate index information for database content not found in the index over time.

Backing up the indexes will be be represented below by the following pseudocode:

backup ${APP}_JFSINDEXES

In the pseudocode above ${APP} represents the application node that needs to be backed up. The code below backs up all nodes.

backup CCM_JFSINDEXES
backup QM_JFSINDEXES
backup JTS_JFSINDEXES

Please note that the RM application does not have its own JFS index files. The JTS maintains the JFS index files for its data which includes the index for the RM application.

JFS Index Files Locations

The location of the JFS index files can be found in the teamserver.properties file for each application as value of the following properties:

com.ibm.team.jfs.index.root.directory

The location of the index files is, by default, relative to the folder the teamserver.properties resides. This should be changed to an absolute path.

Stop Indexing During Backup of Index Files in the 3.x Product Versions

To avoid a corrupted back up of the JFS index files in 3.x products, stop the indexing service during the backup period or shut down the server. Stopping the indexing services can increase the server up time when running backups. It is necessary to resume the indexing services after the backup has been performed.

The indexing service can be stopped using the repotools command -suspendIndexer:

repotools-${APP} -suspendIndexer ${APP}_PARAMETER 

The indexing service can be restarted using the repotools command -resumeIndexer:

repotools-${APP} -resumeIndexer ${APP}_PARAMETER 

In the above pseudocode repotools-${APP} needs to be replaced by the repotools command for the application that is backed up. For example repotools-ccm for the CCM application. In addition ${APP}_PARAMETER represents the required parameters for the command such as repository URL of the Server and the login information for an administrative user. Please refer to the help of the repotools command for the syntax and options. As an example for JTS the command looks like:

repotools-jts -suspendIndexer repositoryURL=https://JTS_SERVER:JTS_SERVER_PORT/jts adminUserID=****** adminPassword=****** 

For simplicity, this article uses the pseudocode below to describe the detailed JFS index back up for the node ${APP}.

suspendIndexing ${APP}_SERVER
backup ${APP}_JFSINDEXES
resumeIndexing ${APP}_SERVER

The code below shows the detailed pseudocode for all nodes.

suspendIndexing CCM_SERVER
backup CCM_JFSINDEXES
resumeIndexing CCM_SERVER

suspendIndexing QM_SERVER
backup QM_JFSINDEXES
resumeIndexing QM_SERVER

suspendIndexing JTS_SERVER
backup JTS_JFSINDEXES
resumeIndexing JTS_SERVER

Backing Up the JFS Index Files in the 4.x Product Versions

In the 4.0 product versions a new repotools command was introduced, that does the back up of the JFS index for an application, with the server being online. Backing up the JFS indexes in 4.x product versions will be described below using the same same pseudocode:

backup ${APP}_JFSINDEXES

Where repotools-${APP} needs to be replaced by the repotools command for the application that is backed up. For example repotools-ccm for the CCM application. In addition ${APP}_PARAMETER represents the required parameters for the command such as repository URL of the Server and the login information for an administrative user. Please refer to the help of the repotools command for the syntax and options. As an example for JTS the command looks like:

repotools-jts -backupJFSIndexes repositoryURL=https://JTS_SERVER:JTS_SERVER_PORT/jts adminUserID=****** adminPassword=****** toFile=FILELOCATION_AND_NAME

Using this command maintains the consistency of the JFS index files and does not require to stop and restart the indexer.

Backup the Fulltext Index Files

Some applications such as CCM and QM also use a Fulltext index to support various search and query operations. You need to back up the Fulltext index files to be able to restore the applications consistently. The Fulltext Index can not be backed up independent of the database.

Backing up the indexes will be be represented below by the following pseudocode:

backup ${APP}_FULLTEXTINDEX

In the pseudocode above ${APP} represents the application node that needs to be backed up. The code below backs up all nodes. JTS does not have a Fulltext index.

backup CCM_FULLTEXTINDEX
backup QM_FULLTEXTINDEX

Ensure a Consistent Backup of Database and Fulltext Index

As described in Defect 245648, it is currently not possible to stop the Fulltext indexer. In order to create a consistent backup of the work item index and to be able to restore from it, it is necessary to backup the Fulltext index files while the respective application is shut down. 

Fulltext Index Files Locations

The location of the Fulltext index files can be found in the teamserver.properties file for each application as value of the following properties:

com.ibm.team.fulltext.indexLocation

The location of the Fulltext index files is, by default, relative to the folder the teamserver.properties resides. This should be changed to an absolute path.

Backup Scenarios

With all relevant backup operations described above, it is now possible to look at different scenarios and provide the sequence of backup operations.

Offline Backup

Offline Backup Sequence for a Single Application on a Single Server.

A complete offline backup procedure for one application on a single server machine would perform the following sequence of activities:

shutdown ${APP}_SERVER
backup ${APP}_CONFIG
backup ${APP}_SERVER_CONFIG
backup ${APP}_JFSINDEXES
backup ${APP}_FULLTEXTINDEX
backup ${APP}_DB
startup ${APP}_SERVER

Offline Backup is the easiest, safest option and easiest to implement option for co located and distributed topologies.

Please note that the RM application does not require backing up of the index files or the database, because this is done during the JTS backup

Offline Backup Sequence for a CLM Solution.

A full offline backup for all nodes could look like

shutdown CCM_SERVER
shutdown QM_SERVER
shutdown RM_SERVER
shutdown JTS_SERVER

backup CCM_CONFIG
backup QM_CONFIG
backup RM_CONFIG
backup JTS_CONFIG

backup CCM_SERVER_CONFIG
backup QM_SERVER_CONFIG
backup RM_SERVER_CONFIG
backup JTS_SERVER_CONFIG

backup CCM_JFSINDEXES
backup QM_JFSINDEXES
backup JTS_JFSINDEXES

backup CCM_FULLTEXTINDEX
backup QM_FULLTEXTINDEX

backup CCM_DB
backup QM_DB
backup JTS_DB

startup CCM-server
startup QM-server
startup RM-server
startup JTS-server

In the scenario above the backup of the JFS Index files assumes suspending the indexer in 3.x and using the backupJFSIndexes command in 4.x.

Offline Backup with Reduced Server Downtime

The example above has the longest downtime for offline backup scenarios. There are several options to reduce downtime, dependent on the systems and infrastructure in use. These include

• Backing up the application configuration files while the server is up
• Backing up the application server configuration while the server is up
• Stopping indexing for backing up the JFS indexes and keeping the server operational during the backup
• Isolating the server from users while backing up the database
• Online backup of the database.

The following example only takes the server offline to backup the Databases and the Fulltext Indexes. 

backup CCM_CONFIG
backup QM_CONFIG
backup RM_CONFIG
backup JTS_CONFIG

backup CCM_SERVER_CONFIG
backup QM_SERVER_CONFIG
backup RM_SERVER_CONFIG
backup JTS_SERVER_CONFIG

backup CCM_JFSINDEXES
backup QM_JFSINDEXES
backup JTS_JFSINDEXES

shutdown CCM-server
shutdown QM-server
shutdown RM-server
shutdown JTS-server

backup CCM_FULLTEXTINDEX
backup QM_FULLTEXTINDEX

backup CCM_DB
backup QM_DB
backup JTS_DB

startup CCM-server
startup QM-server
startup RM-server
startup JTS-server

Online Backup

A continuous online backup is not possible today, due to Defect 245648 and the resulting inability to back up the Fulltext index online. It is necessary to take the server offline at least to back up the Fulltext index. If you need to restore from an online backup without a consistent backup of the Fulltext Index you have to recreate the Fulltext indexes from the restored databases.

Restore From a Backup

Replace backup by restore in the pseudocode to get a restore strategy.

Temporal  Database Consistency During Restore

It is important to keep a valid temporal sequence of the database states during backup and restore. Make sure to restore the databases for all applications registered to a JTS, for example the CCM and the QM database, from a point in time shortly before or equal to the JTS database backup last transaction. This is especially important for using online backup for the database, but also holds for offline backup. In offline backups it is possible to create a consistent backup if all servers are down at the same time. This requirement is due to the JTS creating elements in the registered applications and potential conflicts in case the database of the registered application is ahead of the JTS database.

When restoring the JFS index files from a backup you need to make sure that the time of the backup of the index files is from a point in time before the applications database backup. If you restore a backup of the JFS index files that is more recent than the database of that application, you will see errors indicating that the index is ahead of the database. In this case you need to recreate the index files. 

You also need to restore a backup of the Fulltext Indexs consistent with the databases or you have to recreate the Fulltext index.

A complete restore procedure will perform the following sequence of activities:

restore CCM_CONFIG
restore QM_CONFIG
restore RM_CONFIG
restore JTS_CONFIG

restore CCM_SERVER_CONFIG
restore QM_SERVER_CONFIG
restore RM_SERVER_CONFIG
restore JTS_SERVER_CONFIG

restore CCM_JFSINDEXES
restore QM_JFSINDEXES
restore JTS_JFSINDEXES

restore CCM_FULLTEXTINDEX
restore QM_FULLTEXTINDEX

restore CCM_DB
restore QM_DB
restore JTS_DB

More complex scenarios

With CLM 2011 you have the option to install more than one CCM and QM application on different servers and run it against a common JTS. In that case you must add these additional servers to the backup and perform all backup steps for each individual server. It is necessary to keep the temporal consistency for the JTS and all the applications registered to it. If you use more than one JTS, there is no requirements to keep the temporal consistency between the different groups of JTS and its registered applications. 

Summary

This article described the data and steps required to back up a deployed CLM Solution installation. It provides several options on how to perform a backup and also provides information to perform backups for more complex Jazz deployments.

APPENDIX

Applications Upgraded from 2.x

In the case of an installation that has been upgraded from to 3.0.x the upgraded Applications are using path /jazz instead of /ccm or /qm. Therefore all occurrences of these strings in this article above must be replaced with /jazz.

Recreating the Index Files

It is possible to recreate the index files from a restored database backup. Please be aware that recreation of the index files is an expensive operation and needs to be performed with the server shut down.

Recreate the Fulltext Index

You can recreate the Fulltext index files using the command:

repotools-${APP} -rebuildTextIndices

Recreate the JFS Index

You can recreate the JFS index files using the command:

repotools-${APP} -reindex all

Oracle 

Please see this Technote on Oracle backup especially if intending to use datapump. There is a known issue with Oracle databases that are restored from a backup in 3.0.x. Please see Workitem 180771 for details. The customer support can provide a solution if this issue occurs.

Products upgraded from Version 2.x

In the case of an installation that has been upgraded from a 2.x product version the context root will be jazz instead of ccm. or qm. In these cases all occurrences of the context root used in URL’s and paths above must be replaced with jazz.

---

For more information

• Jazz Foundation Server Administration
• Jazz Administration Plan
• Jazz Deployment Guide
• Considerations for Rational Team Concert 3.x backups
• Tip: Configuring IBM DB2 V9.5 for online backups for Rational Team Concert
• CLM Online Help

---

About the author

Ralph Schoon is a member of the Jazz Jumpstart team. The Jazz Jumpstart team is a worldwide group of development specialists who bring new and advanced Jazz-based technologies to customers. Please direct feedback and comments to ralph.schoon@de.ibm.com

---

Copyright � 2013 IBM Corporation