In version 5.0 the database storage for the RM application changed. RM has now its own database and index files that need to be taken care of in the backup. This has been added to below.
This article has been preliminary updated to include new applications delivered as part of 6.0 as well as the Liberty application server replacement of Tomcat.
Please note, in version 7.x the product names have been changed and each application has been updated to address log4j2. Relevant changes have been made below.
After you start using the IBM solution for Engineering Lifecycle Management (ELM), it will contain more and more important data. Therefore, it is essential to make regular backups of the data to avoid losing it. The Rational solution for Collaborative Lifecycle Management Information Center contains information on how to back up the databases. To minimize down time, you can increase the reliability of the IBM solution for ELM by using hardware that provides high reliability. However, failure can still happen. Since whole development teams depend on the infrastructure, it is desirable to be able to recover it as quickly as possible. This article provides specifics about what to consider when backing up the IBM solution for ELM.
For the purposes of this article, assume that the IBM solution for ELM is deployed in a fully distributed topology.
Note: Since 5.0 the RM Application has its own database RM_DB that needs to be backed up.
Additional ELM applications such as Design Manager and Rhapsody Model Manager require backup like any other Jazz application, if deployed.
In case a data warehouse such as Rational Reporting for Development Intelligence or Rational Insight is set up, keep in mind that setting up a separate backup procedure for its database and configuration data is also needed.
This section answers the question, "What could happen?" In the topology described above, you must plan for the following potential failure scenarios:
The data needed to restore a complete deployment as described above consists of:
A backup procedure is worthless if data is missing, inconsistent, unreadable, or not working for other reasons. Practice and test your backup and restore procedure on an isolated test infrastructure on a regular basis, to be sure that it works.
For normal operations as well as for backup and restore, it is important that the clocks of your machines that run the IBM solution for ELM are closely synchronized. The applications that require the synchronization provide setting up a connection to a NTP server to monitor the machine clocks in the advanced server properties.
While installing the products, you choose install directories. For example, if you installed by using IBM Installation Manager on Linux, the default server installation directory is /opt/IBM/JazzTeamServer
. If you installed on Windows, the default server installation directory is C:\Program Files(X86)\IBM\JazzTeamServer
. This installation directory will be referred to as server-install-dir in the following text. The image below shows a typical folder structure after installing all applications on Apache Tomcat.
During the setup and configuration of Jazz Team Server and the other applications, several files are changed either automatically or manually. Some files that are updated during administration operations and should therefore be included in a regular backup as well.
In this article, the backup of the Jazz application configuration files will be represented by the following pseudocode:
backup APP_CONFIG
In the pseudocode, APP, represents the application node that must be backed up. In the pseudocode for a specific application, APP must be replaced with the acronym that represents the application, which will be JTS, CCM, QM, RM or SUP, where SUP stands for the other applications. This pattern is kept in the article. The pseudocode below would do the backup for all applications:
backup SUPP_CONFIG backup CCM_CONFIG backup QM_CONFIG backup RM_CONFIG backup JTS_CONFIG
All the Jazz application configuration files are stored in one or more folders underneath the configuration data folder:
server-install-dir/server/conf
The options for backing up the Jazz application configuration files are a full backup and selective backup.
You can back up the whole folder structure underneath:
server-install-dir/server/conf
for each application. This option will safely back up all data that might be necessary to restore the Jazz application configuration. On the other hand, it backs up more data than is actually needed.
Note: By default, the index files for the JFS index and the Fulltext index are also stored in this location. If you use a full backup for the folder, make sure that the backup of the index files is successful or change the server configuration and place the index files into a location outside of the server-install-dir/server/conf
folder. See below for more information.
The configuration folder also contains program code which is often not desired to be included in backups. To be more selective, it is possible to only include the most important files and folders into the back up. This can especially exclude all data that is installed with the product.
Each application has its own configuration folder. An application can use more than one folder. The folders must be backed up together.
JTS uses the following folders for JTS and the Lifecycle Project Application (LPA):
server-install-dir/server/conf/jts server-install-dir/server/conf/admin
The CCM application stores its data in the folder:
server-install-dir/server/conf/ccm
The QM application stores its data in the folder:
server-install-dir/server/conf/qm
The RM application stores its data in the folder:
server-install-dir/server/conf/rm
Each application has its own configuration files as shown below:
For each application, at least back up the following files from the configuration folder:
teamserver.properties log4j.properties
Beginning with ELM 7.0.2 SR1 (iFix015) and ELM 7.0.1 SR1 (iFix018) - log4j.properties file has been changed to log4j2.xml, on those servers you should backup the following files from the configuration folder:
teamserver.properties log4j2.xml
In general, back up all files of this format:
*.properties
Some applications store data in RDF format. Check for files of the following format and back them up in case there are any:
*.rdf
Other files created during the installation are typically not modified. If in doubt, check the modification date and time, and include files that changed after the installation finished.
As mentioned above, if you use this as a general advice for Jazz based solutions; for example, to back up Design Manager, or for newer versions of the tool, back up all of the relevant data in all available configuration folders in:
server-install-dir/server/conf
Like JTS, Design Manager uses more than one configuration folder; in this particular case, dm
and vvc
. Be sure to catch all relevant data from these folders. Other applications add their own folders here. For example
am
- Rhapsody Model Manager
dcc
- Data Collection Component
gc
- Global Configuration Management application
ldx
- Link Indexer; see the documentation about backing up the data
lqe
- Lifecycle Query Engine; See the documentation about backing up the data
relm
- Engineering Lifecycle Manager
rs
- Jazz Reporting Service
All these applications behave in similar ways and have configuration and potentially other data such as their own database to back up. JRS also provides a mechanism to export the custom reports.
Do not add any folders to the
server-install-dir/server/conffolder. Especially do not create copies of the existing folders and store them in this folder. This will cause issues with your application; for example, if you run setup again in an upgrade scenario.
A full backup contains all the required data. If you deployed any custom server extensions, and do a selective backup, either back up the corresponding files or at least have the files available for rapid redeployment. The files that are needed to deploy custom server extensions are typically stored in the folder:
server-install-dir/server/conf/APP/provision_profiles
The files that must be backed up are the provision profile .ini
file and the corresponding update site
folder.
A full backup contains all the required data. If you deployed any patches, and do a selective backup, either back up the corresponding files or at least have the files available for rapid redeployment. See the patch description for the files that need to be included in a backup.
Back up the application server configuration data for all application servers, in order to be able to restore it later. How to back up this data and which data needs to be included in the backup depends on the application server in use. The application server configuration changes only due to administrative tasks. It should be sufficient to use an incremental backup once a day.
Backing up the application server configuration files for the APP application will be be represented below by the following pseudocode:
backup APP_APPSERVER_CONFIG
The following pseudocode shows the steps to back up all application server configuration files for all nodes:
backup SUPP_APPSERVER_CONFIG backup CCM_APPSERVER_CONFIG backup QM_APPSERVER_CONFIG backup RM_APPSERVER_CONFIG backup JTS_APPSERVER_CONFIG
When installing on Tomcat, the application server configuration data is in
server-install-dir/server/tomcat/conf/. Some of these files are modified during the setup process and during operation and should be included in the backup procedure.
The following files are typically modified during the setup and should be backed up:
server-install-dir/server/tomcat/conf/server.xml server-install-dir/server/tomcat/conf/web.xml
If you provide a server certificate, include it in your backup. You can find information about the storage location of the used certificate in the server.xml
file. The Connector element contains a keystoreFile property that contains the file location.
If you edited the application's web.xml
files; for instance, when using Tomcat with LDAP, include these files in your backup. The web.xml
files to backup are stored in this location:
server-install-dir/server/tomcat/webapps/APP/WEB-INF/web.xml
For example:
server-install-dir/server/tomcat/webapps/jts/WEB-INF/web.xml server-install-dir/server/tomcat/webapps/ccm/WEB-INF/web.xml
If you are using Tomcat for user authorization, also back up the user registry file:
server-install-dir/server/tomcat/conf/tomcat-users.xml
When using WebSphere Application Server, back up the profile used to deploy the application. See the WebSphere Application Server Knowledge Center for your version of the WebSphere Application Server for more information. One option is to use the manageprofiles command.
When installing on WebSphere Liberty, the application server configuration data is in these folders
server-install-dir/server/liberty/servers/clm/ server-install-dir/server/liberty/servers/clm/conf/
Some of the files underneath these folders are modified during the setup process and during operation and should be included in the backup procedure.
The following files are typically modified during the setup and should be backed up:
server-install-dir/server/liberty/servers/clm/server.xml server-install-dir/server/liberty/servers/clm/conf/ldapUserRegistry.xml server-install-dir/server/liberty/servers/clm/conf/application.xml
If you are using the built in local user authorization, also back up the user registry file:
server-install-dir/server/liberty/servers/clm/conf/basicUserRegistry.xml
If you provide a server certificate, include it in your backup. You can find information about the storage location of the used certificate in the server.xml
file. The element keyStore contains a location property that contains the file location. If this shows a file or a relative path the default location for the file is in
server-install-dir/server/liberty/servers/clm/conf/resources/security/
It is necessary to backup all databases used by the applications. The RM application since version 5.0 has its own database. In versions before 5.0 RM used the JTS database to store its data and therefore did not have a separate database to back up. If you use a data warehouse, also set up a backup procedure for its database.
Backing up the application database for the APP application will be be represented below by the following pseudocode:
backup APP_DB
The pseudocode below represents all steps and the sequence needed to back up for all databases for all applications:
backup SUP_DBs backup CCM_DB backup QM_DB backup RM_DB backup JTS_DB
When backing up a database, make sure that the database backup is complete and consistent from a transaction perspective. A backup is consistent if taken offline, while the database is inactive, or when it is locked for write operations. Some enterprise databases allow consistent online backups with transaction logging or vendor specific procedures and tools.
There are several databases to back up and the applications interact. JTS creates elements in the registered applications. The JTS cannot create an element if it is already there because the database of the registered application is ahead of the JTS. To avoid conflicts and ensure consistency, the databases for all registered applications must be backed up to a point in time equal or earlier than the JTS database. In the example topology, this holds for the CCM and the QM databases. This has an impact on the database restore operation, especially for online backups.
If a Derby database is used, the application server must be shut down to perform the database backup.
To back up the databases, compress and back up the directory and its content used by Derby as repository. By default, the directory is called repositoryDB. The default location for this directory is the derby
folder in the configuration folder of each application:
server-install-dir/server/conf/ccm/derby/repositoryDB server-install-dir/server/conf/qm/derby/repositoryDB server-install-dir/server/conf/rm/derby/repositoryDB server-install-dir/server/conf/jts/derby/repositoryDB
Verify the location of your setup by checking the com.ibm.team.repository.db.jdbc.location
property in the teamserver.properties
file of each application.
See the system requirements on the Getting Started tab of the individual product download page for the supported database management systems. Also see the information center and the library for information about how to backup other databases. For detailed information on how to backup your database, see the documentation for the database management system you are using.
In general, the options provided by enterprise database management systems are:
It is always possible to do an offline backup. Before you back up the database, stop the application server so that no user interaction occurs during the backup. This ensures the backup is consistent. You can also consistently back up all other data related to the server while it is shut down.
Some database management systems, such as DB2, support online backup. This can be used to continuously back up the databases. For more details about DB2 online backup, see this article.
If using online backup, you still want to back up the Jazz application configuration files as well as the application server configuration files. You will most likely not be able to back up and restore the configuration files consistently with the online backup. However, these files are only changed when configuring the application server or administrating the capabilities of the server configuration. Back up these files after performing any of these tasks. Otherwise you must redo changes that did not make it into the last backup.
You also must have a consistent backup of the JFS index and the Fulltext index files. Since version 4.0.5 the JFS index provides a capability to back up the work item index along withe the JFS index. See Fulltext index for considerations, especially for online backup. For backing up DNG indices, see Backup and restoration of Jazz Foundation Service index files for Rational DOORS Next Generation
If a consistent online backup of the work index is the primary concern, A complete offline backup is still the preferred option to get a fully consistent backup of the databases and the other files.
The applications use JFS index files for various query and search operations. These files must be backed up. The JFS index files can be backed up independent from the database. See Query, Search and indexing technologies in CLM and Indices storage and management: Backup, recovery and recreation for more information on the indexes.
To create a consistent backup of the JFS index files and restore from it, back up the JFS index files before or during the backup of the database. The indexing service will re-create index information for database content not found in the index over time. Backing up the indexes will be be represented below by the following pseudocode:
backup APP_JFSINDEXES
In the pseudocode above, APP represents the application node that must backed up. The following code backs up all nodes:
backup CCM_JFSINDEXES backup QM_JFSINDEXES backup RM_JFSINDEXES backup JTS_JFSINDEXES
Note: Up to version 5.0 the RM application did not have its own JFS index files and these indexes where maintained by JTS.
The location of the JFS index files can be found in the teamserver.properties
file for each application as value of the following properties:
com.ibm.team.jfs.index.root.directory
The location of the index files is, by default, relative to the folder where the teamserver.properties
file is located. This should be changed to an absolute path.
To avoid a corrupted backup of the JFS index files in 3.x products, stop the indexing service during the backup period or shut down the server. Stopping the indexing services can increase the server up time when running backups. It is necessary to resume the indexing services after the backup has been performed.
The indexing service can be stopped by using the -suspendIndexer repotools command:
repotools-APP -suspendIndexer APP_PARAMETER
The indexing service can be restarted by using the -resumeIndexer repotools command:
repotools-APP -resumeIndexer APP_PARAMETER
In the above pseudocode, repotools-APP needs to be replaced by the repotools command for the application that is backed up. For example, repotools-ccm
for the CCM application. In addition, APP_PARAMETER represents the required parameters for the command, such as the repository URL of the server and the login information for an administrative user. For the syntax and options, see the repotools command help. As an example for JTS, the command looks like this:
repotools-jts -suspendIndexer repositoryURL=https://JTS_SERVER:JTS_SERVER_PORT/jts adminUserID=****** adminPassword=******
For simplicity, this article uses the pseudocode below to describe the detailed JFS index backup for the APP node:
suspendIndexing APP_SERVER backup APP_JFSINDEXES resumeIndexing APP_SERVER
The following code shows the detailed pseudocode for all nodes:
suspendIndexing CCM_SERVER backup CCM_JFSINDEXES resumeIndexing CCM_SERVER suspendIndexing QM_SERVER backup QM_JFSINDEXES resumeIndexing QM_SERVER suspendIndexing RM_SERVER backup RM_JFSINDEXES resumeIndexing RM_SERVER suspendIndexing JTS_SERVER backup JTS_JFSINDEXES resumeIndexing JTS_SERVER
In version 4.0, a new repotools command was introduced that does the backup of the JFS index files for an application, with the server online.
Backing up the JFS index in version 4.0. and later is described by using the same pseudocode:
backup APP_JFSINDEXES
Where repotools-APP must be replaced by the repotools command for the application that is backed up. For example, repotools-ccm
for the CCM application. In addition, APP_PARAMETER represents the required parameters for the command, such as the repository URL of the server and the login information for an administrative user. For the syntax and options, see the repotools command help. As an example for JTS, the command looks like this:
repotools-jts -backupJFSIndexes repositoryURL=https://JTS_SERVER:JTS_SERVER_PORT/jts adminUserID=****** adminPassword=****** toFile=FILELOCATION_AND_NAME
Using this command maintains the consistency of the JFS index files and does not require to stop and restart the indexer.
Note: In versions before 4.0.5 the defect Defect 245648 prevents from suspending indexing the Fulltext index. In these versions it is necessary to do a consistent backup of these indexes, while the server is running.
Since version 4.0.5 the JFS index provides a capability to back up the work item index along with the JFS index. See Fulltext index for considerations, especially for online backup.
Some applications, such as CCM and QM, also use a Fulltext index to support various search and query operations. You must back up the Fulltext index files to be able to restore the applications consistently. The Fulltext index cannot be backed up independent of the database.
Backing up the indexes is represented by the following pseudocode:
backup APP_FULLTEXTINDEX
In the pseudocode above, APP represents the application node that needs to be backed up. The code below backs up all nodes. JTS does not have a Fulltext index:
backup CCM_FULLTEXTINDEX backup QM_FULLTEXTINDEX
Beginning with version 4.0.5 the work item index backup is now included in the back up of the JFS index.
However, the work item index is still built up during work item operations. If restored from an online backup of the JFS index, it does not index operations that are not included in the backup. So other than the rest of the JFS index files, the work item index does not catch up with later changes and will remain inconsistent with later work item operations. Note as of 5.0.1, the fulltext work item index does catch up with the database similarly to how the JFS indices catching up.
To get a consistent backup of the work item index files a recreation of the index might still be necessary. If a consistent online backup of the work index is the primary concern, a complete offline backup should then be performed in order to get a fully consistent backup of the databases and the other files.
The location of the Fulltext index files can be found in the teamserver.properties
file for each application as value of the following properties:
com.ibm.team.fulltext.indexLocation
The location of the Fulltext index files is, by default a relative path. It will be based on the application relative runtime location. If your CLM application is running on Tomcat this will be the same as the application configuration directory; if your CLM application is running on WebSphere Application Server, this will be the profile path, for example: /
This should be changed to an absolute path.
You can export the reports from the report service and also import them in a different system. This is especially interesting when using backup and restore as a means to create test systems.
In a CCM clustered configuration backup the database and at least one of the cluster nodes with the configuration and index data. To restore use that node as template to create additional nodes.
With all relevant backup operations described above, it is now possible to look at different scenarios and provide the sequence of backup operations.
If you shut down an ELM environment that uses multiple distributed servers, make sure to
You can start up and shut down the other applications in any order, if you make sure the JTS is handled as described above.
A complete offline backup procedure for one application on a single server machine would perform the following sequence of activities:
shutdown ${APP}_SERVER backup ${APP}_CONFIG backup ${APP}_SERVER_CONFIG backup ${APP}_JFSINDEXES backup ${APP}_FULLTEXTINDEX backup ${APP}_DB startup ${APP}_SERVER
Offline backup is the easiest, safest option and easiest to implement option for co-located and distributed topologies.
Note: Since version 5.0 The RM application does require backing up of the index files or the database. In versions before this was done with the JTS backup
A full offline backup for all nodes could look like this example:
shutdown CCM_SERVER shutdown QM_SERVER shutdown RM_SERVER shutdown JTS_SERVER backup CCM_CONFIG backup QM_CONFIG backup RM_CONFIG backup JTS_CONFIG backup CCM_SERVER_CONFIG backup QM_SERVER_CONFIG backup RM_SERVER_CONFIG backup JTS_SERVER_CONFIG backup CCM_JFSINDEXES backup QM_JFSINDEXES backup RM_JFSINDEXES backup JTS_JFSINDEXES backup CCM_FULLTEXTINDEX backup QM_FULLTEXTINDEX backup CCM_DB backup QM_DB backup RM_DB backup JTS_DB startup JTS-server startup CCM-server startup QM-server startup RM-server
In the scenario above the backup of the JFS index files assumes suspending the indexer in 3.x and using the backupJFSIndexes command in 4.x. During startup in a distributed environment, start JTS first, then start the other servers.
The example above has the longest downtime for offline backup scenarios. There are several options to reduce downtime, dependent on the systems and infrastructure in use. These include:
The following example only takes the server offline to backup the databases and the Fulltext index files:
backup CCM_CONFIG backup QM_CONFIG backup RM_CONFIG backup JTS_CONFIG backup CCM_SERVER_CONFIG backup QM_SERVER_CONFIG backup RM_SERVER_CONFIG backup JTS_SERVER_CONFIG backup CCM_JFSINDEXES backup QM_JFSINDEXES backup RM_JFSINDEXES backup JTS_JFSINDEXES shutdown CCM-server shutdown QM-server shutdown RM-server shutdown JTS-server backup CCM_FULLTEXTINDEX backup QM_FULLTEXTINDEX backup CCM_DB backup QM_DB backup RM_DB backup JTS_DB startup JTS-server startup CCM-server startup QM-server startup RM-server
As of 5.0.1, a continuous online backup is possible which means both the JFS index and Fulltext index files can be backed up online. However, for versions prior to 5.0.1, it is necessary to take the server offline at least to back up the Fulltext index. In that case, to restore from an online backup without a consistent backup of the Fulltext index, you must re-create the Fulltext index files from the restored databases.
Replace backup by restore in the pseudocode to get a restore strategy.
Keep a valid temporal sequence of the database states during backup and restore. Make sure to restore the databases for all applications registered to a JTS; for example, the CCM and the QM database, from a point in time shortly before or equal to the JTS database backup last transaction. This is especially important for using online backup for the database, but also holds for offline backup. In offline backups, you can create a consistent backup if all servers are down at the same time. This requirement is due to the JTS creating elements in the registered applications and potential conflicts in case the database of the registered application is ahead of the JTS database.
When restoring the JFS index files from a backup, make sure that the time of the backup of the index files is from before the application's database backup. If you restore a backup of the JFS index files that is more recent than the backup of the database of that application, error messages will indicate that the index is ahead of the database. In this case, you must re-create the index files.
You also must restore a backup of the Fulltext index files that is consistent with the databases or you must re-create the Fulltext index. A complete restore procedure will perform the following sequence of activities:
restore CCM_CONFIG restore QM_CONFIG restore RM_CONFIG restore JTS_CONFIG restore CCM_SERVER_CONFIG restore QM_SERVER_CONFIG restore RM_SERVER_CONFIG restore JTS_SERVER_CONFIG restore CCM_JFSINDEXES restore QM_JFSINDEXES restore RM_JFSINDEXES restore JTS_JFSINDEXES restore CCM_FULLTEXTINDEX restore QM_FULLTEXTINDEX restore CCM_DB restore QM_DB restore RM_DB restore JTS_DB
Since CLM 2011, you can install more than one CCM and QM application on different servers and run them against a common JTS. In that case, you must add these additional servers to the backup and perform all backup steps for each individual server. It is necessary to keep the temporal consistency for the JTS and all the applications registered to it. If you use more than one JTS, there is no requirement to keep the temporal consistency between the different groups of JTS and its registered applications.
This article described the data and steps required to back up a deployment of the IBM solution for ELM. The article provides several options for backing up and also provides information about how to back up more complex Jazz deployments.
In the case of an installation that was upgraded to a version of the 3.0 release, the upgraded applications use the /jazz
path instead of /ccm
or /qm
. Therefore, all occurrences of these strings in this article must be replaced with /jazz
.
You can re-create the index files from a restored database backup. Be aware that re-creation of the index files is an expensive operation and must performed with the server shut down.
You can re-create the Fulltext index files by using this command:
repotools-APP -rebuildTextIndices
You can re-create the JFS index files by using this command:
repotools-APP -reindex all
See this technote about Oracle backup, especially if you intend to use datapump.
There is a known issue with Oracle databases that are restored from a backup in 3.0.x. For details, see work item 180771. If this issue occurs, Support can provide a solution.
In the case of an installation that was upgraded from a release of version 2, the context root will be jazz
instead of ccm
or qm
. In these cases, all occurrences of the context root in the URLs and paths above must be replaced with jazz
.
Status icon key: