TN0006: Deploying Rational Team Concert into a ClearCase/ClearQuest Environment
Summary
This document provides guidance for deploying Rational Team Concert into an existing ClearCase/ClearQuest environment. In particular, it provides guidance for deploying the ClearCase and ClearQuest Synchronizers to synchronize the information stored in Rational Team Concert and ClearCase/ClearQuest.
Rational Team Concert also includes bridges to ClearQuest and ClearCase which allow you to associate ClearQuest records or source control information with work items in the Jazz repository, without synchronizing data.
For information on setting up the Rational ClearQuest Bridge, see Configuring and using the Rational ClearQuest bridge in the Rational Team Concert Information Center.
For more information on the Rational ClearCase Bridge, see Using the ClearCase Bridge in the Rational Team Concert Information Center.
ClearCase Synchronizer
The primary deployment principle for the ClearCase synchronizer is that you should configure your ClearCase and Rational Team Concert deployments independently, i.e. configure ClearCase for the ClearCase users (ignoring Rational Team Concert) and configure Rational Team Concert for the Rational Team Concert users (ignoring ClearCase). Then deploy the ClearCase Synchronizer on a machine with efficient read/write access via a ClearCase dynamic view to the ClearCase stream/branch that is to be synchronized with a Rational Team Concert stream.
ClearCase Multisite Considerations
You should not modify your ClearCase Multisite deployment because of the addition of Rational Team Concert, and you should not modify your Rational Team Concert deployment because of the use of ClearCase Multisite. When deploying the ClearCase Synchronizer, just follow the standard guidance above. In particular, since the Synchronizer must have write access to the ClearCase stream/branch, this means that the ClearCase Synchronizer for that ClearCase stream/branch must be deployed at a site where that stream/branch is mastered.
Selecting the Teams that should use Rational Team Concert
A team that is currently using (or is considering using) a source control system such as Subversion or Perforce is a good candidate for using Rational Team Concert. Ideally, developers on the team would be Eclipse or Visual Studio users. Command line access is provided for other users
Assigning a Software Component to a Repository
When an organization uses both ClearCase and Rational Team Concert, each software component (set of files forming a logical unit of software development) should be assigned to a particular software repository (Jazz repository, or ClearCase VOB). The repository to which a software component is assigned should be based on what repository is being used by the team that most commonly modifies that software component. In particular, If a team is using Rational Team Concert, that team should be assigned to a single Jazz repository, to ensure that the team can benefit from all of the collaborative capabilities of Rational Team Concert, and then all software components owned by that team should be assigned to the Jazz repository for that team.
Read-Only Access to ClearCase files by a Rational Team Concert User
When a team using Rational Team Concert needs to have read-only access to files that are assigned to a ClearCase VOB, they should create a ClearCase view that selects the desired configuration of those files. Since this is read-only access, a single view can be logically shared by all of the developers that want to see that configuration.
Write Access to ClearCase files by a Rational Team Concert User
When a Rational Team Concert user needs write access to files that are assigned to a ClearCase VOB, they have two choices:
- Create a personal ClearCase view, and checkin changes to the files in that view. Since Eclipse supports concurrent use of multiple SCM providers in a single Eclipse workspace, it is relatively straightforward for a user to change both files in a Jazz repository workspace and files in a ClearCase view in a single Eclipse workspace.
- Use the “ClearCase Synchronizer” to create a “ClearCase synchronized stream” in their Jazz repository, where a ClearCase synchronized stream is a Jazz stream that is logically associated with a user-specified ClearCase stream or branch-type (to avoid excessive wordiness, we will use simply “ClearCase stream” to mean “ClearCase stream or branch-type” and use simply “synchronized stream” to mean “ClearCase synchronized stream”). This replicates user-selected files from the specified ClearCase stream into the synchronized stream in the Jazz repository, and allows the user to use the Jazz SCM tooling to modify both the files from their Jazz repository and the (replicated) files from the ClearCase repository. A synchronized stream is automatically “synchronized” periodically (commonly nightly) with its associated ClearCase stream. When a synchronized stream is synchronized, any changes that have been delivered to the synchronized stream are applied to the ClearCase stream, and any changes that have been checked in to the ClearCase stream are applied to the synchronized stream. Any conflicts resulting from concurrent changes in Rational Team Concert and ClearCase to the same file or directory will automatically be detected, and the standard Jazz merge tooling is then used to resolve those conflicts.
The primary considerations in deciding whether a Rational Team Concert user should use a ClearCase view or a ClearCase synchronized stream to modify files in a ClearCase component are:
- The number of files being changed. If the user needs write access to only a moderate number of files in a ClearCase component (a few hundred), use of the ClearCase Synchronizer is probably preferable, because the user does not have to bother with setting up a personal ClearCase view.
- The number of Rational Team Concert users making changes to those files. If multiple Rational Team Concert users are making changes to the same ClearCase component, then the use of the ClearCase Synchronizer is preferable, because the Rational Team Concert users can take advantage of the Rational Team Concert collaborative capabilities while modifying those files.
Selecting a Synchronization Host
The host on which the ClearCase Synchronizer is installed is called the “synchronization host.” The synchronization process that applies changes from ClearCase to the Jazz repository and from the Jazz repository to ClearCase runs on the synchronization host. The synchronization process therefore must be able to read and write data in the Jazz repository and Rational ClearCase VOBs. To enable this, the following conditions must be true:
- The host must have Rational ClearCase installed, and must have a hardware and network configuration that provides good dynamic view performance.
- Because the ClearCase Synchronized Streams managed by a given synchronization host should be created on that host, the host must have the Rational Team Concert client installed.
- If you plan to synchronize with UCM projects that are integrated with Rational ClearQuest, the synchronization host must have a Rational ClearQuest client installed. That client must be configured to have access to those Rational ClearQuest databases that the UCM projects are configured to use, and Rational ClearQuest connection properties must be set correctly in the ClearQuest properties in the ClearCase tab of the Synchronized Stream Properties dialog.
- Because time-based rules are used when synchronizing with base Rational ClearCase, the synchronization host and the VOB server hosts it accesses should use a common time source if the ClearCase Synchronized Stream includes files and folders from base Rational ClearCase.
Initial Import
The cost of a synchronize operation is proportional to the number of files that have changed, and proportional to the number of components that contain synchronized files. The amount of time will also vary with the database you have deployed, server on which you are running, etc. An initial import of files from the ClearCase stream to the synchronized stream is logically no different than a subsequent synchronization, except since every file selected has “changed”, the cost of the initial import is proportional to the number of files selected. (Some performance data is gathered in Appendix if you are interested in what to expect.)
In order to verify that there are no problems in the way you have configured your synchronization server, it is best to first synchronize just a few files (or even just one file). Once you have your first successful synchronization, you should then incrementally add file trees (a few hundred files at a time).
For example, suppose you have a ClearCase component containing 2000 files, whose root directory is /vobX/compA. Suppose that compA contains three Eclipse projects, rooted at /vobX/compA/proj1, /vobX/compA/proj2, and /vobX/compA/proj3, where proj1 contains 1500 files, and proj2 and proj3 each contain 250 files. To get started, first just select a single file to be synchronized, for example, /vobX/compA/proj2/src/parser/main.c. If that synchronize succeeds, then you know your synchronization server is correctly configured, so you can now synchronize /vobX/compA/proj2. Once that succeeds, you can request that the rest of the files be synchronized. Since proj3 just contains 250 files, /vobX/compA/proj3 can be specified as a directory to be synchronized. But proj1 contains 1500 files, so it should be imported in several batches. For example, suppose that proj1/src contains five subdirectories, named db, db-core, protocol, repo, and admin, each with about 300 files. First import the file directories: /vobX/compA/proj1/src/db, vobX/compA/proj1/src/db-core, etc. If those five synchronizations succeeds, you can then import /vobX/compA/proj1, and finally, once all of the projects have been successfully imported, you can import simply /vobX/compA.
Once you have successfully synchronized a directory, you can remove any children of that directory from the list of files and directories to be synchronized. So in the above example, once /vobX/compA has been imported, you can remove all of the previous files and directories from the “selected for synchronization” list.
Stream Synchronization
Once you no longer need write access to any of the files in a given synchronized directory, you should remove that directory from the list of directories to be synchronized, so that you no longer pay the cost of synchronizing changes to the files in that directory (and therefore free up synchronization resources for new sets of files you might need to modify).
A directory that has been removed from the list to be synchronized can subsequently be added back, but note that this is effectively a new import, which will overwrite the current state of those files with whatever is the current state of those files in the other repository (and no conflicts will be detected or reported). This means that the change-set that results from adding back a previously synchronized directory should be inspected to ensure that no unexpected overwrites have occurred. Note that these change sets are automatically given a special comment indicating that they should be inspected in this way.
If you find that you only need a subtree under a directory that you have synchronized, you can add the root directory of that subtree as a directory to be synchronized, and then remove the parent directory from the list of directories to be synchronized. Note that for efficiency, it should be done in this order (“add the sub-directory”, then “remove the parent directory”), and not “remove the parent directory”, then “add the sub-directory”), because the latter order will need to re-import all of the changes in that sub-directory, to make sure there haven’t been any changes made in that sub-directory while it was not being synchronized.
Selecting the ClearCase Stream to Synchronize
As indicated in the import section, the cost of a synchronize operation is primarily proportional to the number of files that have changed. A key consideration for which stream in ClearCase with which to synchronize is that although the synchronize stream in Rational Team Concert is always open for additional deliveries, the ClearCase stream is locked while a synchronization is taking place. If only a small number of changes are made each day, and the synchronization is scheduled at a time when nobody is attempting to checkin changes to that ClearCase stream, then one can synchronize with a ClearCase integration stream. But if the lock on the integration stream during synchronization is interfering with normal development on that stream, a new ClearCase development stream should be created for the synchronized stream, and the synchronized stream should be retargeted to that new development stream. It will then be necessary for the user (or some scheduled job) to periodically rebase that development stream, to pick up changes that have been made in ClearCase on the integration stream.
ClearQuest Synchronizer
Teams sometimes want different tools and processes than other teams or the enterprise. Sometimes the needs of teams across the enterprise are aligned but different schedules make it impossible to coordinate an enterprise-wide shift to new tools or processes. Synchronizers enable different teams within an enterprise to make different tool and process decisions while still sharing development assets. In the case of the ClearQuest Synchronizer to Rational Team Concert those assets are work items, e.g. defects, enhancement requests, change requests, tasks, etc. The synchronizer is perfect for situations where some teams want the tight integration of Rational Team Concert but must share those work items with other teams using Rational ClearQuest.
Consider a development team that enjoys the integration of work items with build and source control in Rational Team Concert. Perhaps their support team enters customer reported issues through ClearQuest and the team needs to see them. Or, consider an enterprise that gathers project metrics from a centralized ClearQuest database in which every team must make their work visible.
If a team doesn’t care to use the work item component in Rational Team Concert they could elect to continue using ClearQuest directly. Similarly, a team may be able to access Rational Team Concert work items, perhaps through the web UI, just as easily as ClearQuest.
One of the goals of connecting the two systems is to allow teams some control over their process. This implies that there will not be complete alignment between a process specification’s work item definition and a ClearQuest schema’s record type. Fields, enumeration choices, states, and even the variety of types available may differ. The synchronizer cannot infer the semantics of the two processes or distill the key data to synchronize — this requires someone knowledgeable of both the Jazz process specification and the ClearQuest schema to produce a mapping. The synchronizer captures this mapping in the form of synchronization rules. The more similar the two processes the easier it will be to create synchronization rules, but in some cases this will require significant effort and could involve making tradeoffs and converging aspects of the process definitions.
Item Synchronization
Synchronization rules that map Jazz Work Items to ClearQuest records are scoped to a Jazz Project Area. Project Areas each have their own process specification which defines, among other things, the types of work items, their fields, and state transition diagrams. Since work items may look different in each Project Area the synchronization rules that map to ClearQuest records also need to vary.
Not all work items in a project area need to be synchronized with ClearQuest. The Synchronization Rule editor specifies which Team Areas’ work items should be sent to ClearQuest. Similarly, most often only a subset of ClearQuest records should be sent to a Project Area or Jazz Team Server. The ClearQuest Synchronizer Gateway uses ClearQuest queries to partition records destined for each Project Area. One query per record type is created for each Project Area to define the subset of records sent to that Project Area. If the Project Areas reside on the same server instance, the query result sets will not overlap because a record can only be synchronized to one work item in a Jazz Team Server at a time. Said contrapositively, you can connect a CQ record to multiple Work Items if and only if the Work Items are on different server instances.
An enterprise may deploy several Rational Team Concert “islands” with separate Jazz Team Servers, and each of these may need to connect to the same enterprise ClearQuest database. The single ClearQuest Synchronizer Gateway for that database can be configured to synchronize with multiple Jazz Team Servers. Filtering ClearQuest records by Project Area continues to work just as in the single Jazz Team Server case, but Project Areas that are synchronized with a single ClearQuest database must have unique names regardless of which repository holds them.
Databases replicated with ClearQuest MultiSite can be connected to a Jazz Team Server with some limitations. Only one ClearQuest Synchronizer Gateway may be deployed for a given database family and must be configured to use one replica (the synchronization replica). As in the non-replicated case the ClearQuest Synchronizer Gateway should be deployed close to the database serving the replica. Changes made in other ClearQuest replicas will synchronize to the Jazz Team Server after ClearQuest MultiSite replicates those changes to the synchronization replica. Similarly, all changes originating from the Jazz Team Server will attempt to synchronize with that ClearQuest replica. However, ClearQuest will block attempts to modify records that are not mastered by the synchronization replica. The synchronizer will report these cases just like any other process errors that occur when attempting synchronization. This error will be resolved after mastership of the record is moved to the synchronization replica.
Batch Item Synchronization
The ClearQuest Synchronizer includes an additional application named synctool which can be used to import existing ClearQuest records as Jazz items. Depending on how many fields are being synchronized, it typically takes a few seconds per record to create the corresponding Jazz item.
The batch synchronization tool uses the same ClearQuest queries defined to select records for project areas to determine which records to import. It can only handle a few thousand records at a time, so the queries should be configured to not select very large numbers of records. Note that it typically would not make sense to import ClearQuest records that do not represent new work to be done in Rational Team Concert (e.g. closed records), so one way to restrict the number of existing records imported would be to have the queries only select open records. Then, after doing the batch import, remove the restriction to open records for normal synchronization, so that even after a record is closed, any changes continue to be synchronized with the corresponding Jazz items.
Batch synchronization should only be attempted after synchronization rules have been defined and successful synchronization has been demonstrated for a few records. Otherwise, many errors could occur if run against a large number of records when there is a problem with the synchronization configuration.
Appendix 1: Synchronization Performance Data of ClearCase Synchronizer 4.0
This section provides some idea of the times to expect for running an initial import and the times to expect to synchronize changes. This is by no means a benchmark, but hopefully it will give some idea of what to expect when using the synchronizer.
The tests were run in the following environment:
- Jazz server (Rational Team Concert 4.0; Repository DB: DB2; Tomcat)
- Sync host and the VOB server are on the same host
- Both sync host and Jazz server are Windows machine: Windows Server 2008, 64-bit Operating System; CPU 3.66 GHz (4 processors); RAM 8 GB
- ClearCase version 8.0.0
Initial Synchronization Import
Number of files imported* || Total time for import to finish || Time per file to import (seconds) 118 1m 25s 0.72 230 2m 11s 0.57 342 2m 33s 0.46 907 5m 35s 0.35 1472 11m 53s 0.48 3841 27m 58s 0.44 6219 44m 57s 0.43 10,047 1h 9m 47s 0.42 16,263 1h 50m 51s 0.41 56,468 6h 34m 54s 0.42 65,129 7h 34m 33s 0.42 138,608 15h 37m 0.41 203,737 24h 58m 24s 0.44 *Number of files imported: The number of files/folders which are contained in the single sync root.
Update Synchronization
Number of changes to sync* || Total time for the sync to finish || Time per change to sync (seconds) 20 4 m 10s 12.5 100 4 m 25 s 2.65 200 5 m 21 s 1.61 300 6 m 12 s 1.24 600 7 m 4 s 0.71 790 7 m 21 s 0.55 1300 10 m 56 s 0.51 *Number of changes to sync: The change on ClearCase side is about 10~20 bytes addition per file; Sync root folder contains 1500 files/folders; ClearCase vob component contains 200K files/folders.
Appendix 2: Synchronization Performance Data of ClearCase Synchronizer 3.0.1
The tests were run in the following environment:
- Jazz server (Rational Team Concert 3.0.1, Repository DB: DB2; Tomcat)
- Sync host and the VOB server are on the same host
- Both sync host (RTC client) and Jazz server are Windows machine: Windows Server 2008, 64-bit Operating System; CPU 3.66 GHz (4 processors); RAM 8 GB
- ClearCase version 7.1.2
Initial Synchronization Import
Number of files imported* || Total time for import to finish || Time per file to import (seconds) 112 1 m 32 s 0.82 557 6 m 46 s 0.660 1453 15 m 35 s 0.653 3797 43 m 30 s 0.687 6140 1 h 3 m 55 s 0.625 16,072 2 h 33 m 58 s 0.575 48,292 8 h 32 s 0.636 112,681 20 h 35 m 0.658 209,047 36 h 56 m 0.636 *Number of files imported: The number of files/folders which are contained in the single sync root.
Update Synchronization
Number of changes to sync* || Total time for the sync to finish || Time per change to sync (seconds) 20 6 m 18 100 6 m 20 s 3.8 200 7 m 14 s 2.17 300 8 m 10 s 1.63 600 10 m 34 s 1.06 700 10 m 45 s 0.921 1100 12 m 1 s 0.655 *Number of changes to sync: The change on ClearCase side is about 10~20 bytes addition per file; Sync root folder contains 1500 files/folders; ClearCase vob component contains 200K files/folders.
References
The following documents contain information relevant to this topic:
- Jazz concepts for Rational ClearCase users – Rational Team Concert 3.0 Online Help
- Configuring and using the Rational ClearQuest Synchronizer – Rational Team Concert 3.0 Online Help
Copyright © 2011, 2012 IBM Corporation