Query and Search indices management in the Rational solution for Collaborative Lifecycle Management
Part 2: Indices storage and management: Backup, recovery and recreation

Jorge A. Diaz, Jazz Jumpstart Team
Contributions from: Martin Aeschlimann, Philippe Mulet, John Whitfield, Simon Helsen, Jerome Lanneluc
Last updated: January 31, 2013
Build basis: Rational solution for Collaborative Lifecycle Management 2012 – Rational Team Concert, Rational Quality Manager, Rational Requirements Composer 4.0.1

The purpose of this series of articles is to provide a basic understanding of the indexing processes in Jazz used by information query and search features, the technologies involved and to provide guidance on the associated administering tasks. We will also briefly review some details on the base architectural details, and how information is stored and queried in the different CLM applications.

This second part of the article series will discuss different administering tasks related with indices management and indices storage details.

Other articles in this series:

• Part 1: Query, Search and indexing technologies in CLM

---

For the remainder of this article is important to keep in mind what querying and indexing technology is applicable for each CLM application. You can find detailed information in the Part 1 of this article, summarized in the section called “Recap: Search and indexing in your CLM deployment”.

In this article we will cover the storage recommendations and administration tasks for maintaining the indices used by JFS and FullText services. The Item Query Service is not relevant for this discussion as the query and indices features are supported by the database.

Indices Storage

Indices storage location

The storage of the indices is determined by a couple of application configuration properties, one for each type of index (JFS or FullText). These properties appear in the teamserver.properties file with the following default values after initial application post-installation setup:

com.ibm.team.jfs.index.root.directory=indices
com.ibm.team.fulltext.indexLocation=conf/<APP>/indices/workitemindex

For example in a typical CCM application deployment, <APP> would be “ccm”.

These teamserver.properties file entries have the corresponding parameter options in the Advanced Properties wizard for each application, which is accessed through the url of the form:
https://<server>:<port>/<contextRoot>/admin#action=com.ibm.team.repository.admin.configureAdvanced

• JFS index
  
• FullText index
  

Keep in mind that all CLM applications but RM have a “teamserver.properties” file for the application configuration parameters such as database location, and the storage of JFS and FullText indices. RM uses JTS for storage, querying and indexing. Note also that although all these properties exist in all the “teamserver.properties” files, the actual relevance of them in a particular CLM application will depend on the query and search technologies being used. Refer to Part 1 of this article for this mapping information between CLM applications and indexing technologies in use.

How are those configuration properties used to determine the storage on disk?

• If using an absolute path value to specify the location of the indices storage, this path location will be used.
• If using relative path (as the default configuration shown earlier), the behavior will be:
  • JFS index (“com.ibm.team.jfs.index.root.directory” property): the actual path will be based on the application configuration directory. For example, C:IBMJazzTeamServerserverconf<app>
  • FullText (“com.ibm.team.fulltext.indexLocation” property): in this case, the contents location path will be based on the application relative runtime location. If your CLM application is running on Tomcat this will be the same as the application configuration directory; if your CLM application is running on WebSphere Application Server, this will be the profile path, for example:
    /<WAS_ROOT>/profiles/AppSrv01/

The following screenshot shows the layout of a CCM application deployment using Tomcat/Derby and the default configuration:

This storage location can be modified by changing the value of the reviewed properties: using the Advanced Properties wizard (recommended), or by modifying the application’s “teamserver.properties” file. Changing any of these values need the application to be restarted to take effect. Such a storage location change will create the empty folders to store the new indices contents, but no old indices will be migrated automatically: you will need to copy over from the old location the indices with the application shut down or perform a reindex to regenerate them in this new location.

Indices storage recommendations

We recommend the following configurations for indices storage management:

• Storage location definition: when deploying your CLM solution, it is strongly recommended to modify its indices configuration specifying a full path location. Moreover, placing the indices in a location different from the default one it’s preferred so they don’t get accidentally deleted for example, when uninstalling after an upgrade. For your enterprise deployment it is recommended to plan your installation storage layout with the indices storage in mind.
• Local vs. network location: general recommendation is to keep indices local to the servers so performance is optimal
• Network storage location: if using a network storage, the ideal characteristics of such storage are:
  • Try to avoid connection over ethernet: in case of connectivity disruptions it may end up resulting in indices corruption
  • Low latency connection link connection advised (e.g, SAS, Fibre Channel or InfiniBand)
  • If possible try to use a file system that has capabilities for HA and redundancy(e.g, SAN or RAID 10)

Indices in a CLM High Availabitity deployment

Enterprise level deployments usually consider one of the possible HA configurations for the CLM solution. We will review some considerations for the inexing storage in these configurations:

CLM clustering: each CLM application node will have its own copy of the indices, its own storage defined, no sharing of indices. The indices information will be kept synchronized using the clustering information sync features.

Standby configurations for failover:

---

Indices Administration Tasks

This section of the article will high-light some of the typical administration tasks that you may need to perform for the indices maintenance.

Backup and Recovery

Given the importance that the indices have for querying and searching for information, it is crucial that you consider the indices backup and recovery procedures as part of your general CLM backup strategy. The backup of the indices should be taken along with the database to ensure information consistency: to have an snapshot of database information and indices content. For this backup process however, we need to differentiate how the indices in play differ in nature:

• JFS indices: repository tools commands are available to perform a backup of these files automatically. An example of such command for performing the backup of JFS indices for the Jazz Team Server application would be as follows:
  

  repotools-jts -backupJFSIndexes repositoryURL=https://JTS_SERVER:JTS_SERVER_PORT/jts adminUserID=****** adminPassword=****** toFile=FILELOCATION_AND_NAME

• Fulltext index: there is no way of ensuring a consistent backup of the files of this indexing technology without performing a server shutdown. Therefore, to make sure that the indices are consistent with database information, the application should be first stopped/shut down to perform the backup copy. Note that this type of indices are updated synchronously on information update, so failing to stop the application for performing the backup could result in a corrupted copy or an invalid backup because of lost information events.

In spite of the possibility of performing an online backup of the JFS indices, given the restrictions that Fulltext indices impose for CCM and QM applications, it is advised to backup both indices when the CLM applications are shut down. For complete information on backup please check the jazz.net article Backup the Rational solution for Collaborative Lifecycle Management.

Similarly, the recovery of indices should be performed along with the application repository recovery to ensure consistency of the information. Note that is particularly important for the FullText indices given how its contents are updated: a missed information update between repository contents and indices information will require you to perform an indices recreation. JFS indices would be able to recover nicely if information in database is ahead although a sync of both is still desired to avoid queries information discordancies and performance impact while information catches up.

Recreation of indices

The indices can be recreated. The situations in which you will have to consider indices recreation are:

• In case of system recovery if no proper backup exists
• In case of indices storage failure
• In situations where where you are instructed to recreate them by Support, or by the official product documentation of another administrator task.

Repository tools commands are available to perform this recreation, having different commands for the different type of indices.

• JFS indices recreation: repotools-<app>.bat -reindex all. Note the importance of the “all” parameter, without it just a subset of the indices would be recreated. Check complete command syntax details here.
  An example of recreating JFS indices for JTS application:
  

  repotools-jts.bat -reindex all

• Fulltext indices recreation: repotools-<app>.bat -rebuildTextIndices. Remember that this recreation is only useful for CCM and QM applications, which are the ones where this indexing technology is used.
  Check complete command syntax details here.
  An example of recreating Fulltext indices for CCM application:
  

  repotools-ccm.bat -rebuildTextIndices

Note that both commands need the server to be shut down before executing them. The time for recreating the indices can take long depending on how big your repository is.
The example commands shown are for Windows platform based deployments. Corresponding commands for Unix/Linux platform deployments exist. Check the official documentation in the provided Information Center.

---

For more information

• Backup the Rational solution for Collaborative Lifecycle Management
• Repotools Command Reference for rebuilding JFS indices at InfoCenter
• Repotools Command Reference for rebuilding Fulltext indices at InfoCenter
• Jazz work item 245648: Backing_up work item indices require a server shutdown

Copyright © 2013 IBM Corporation