Guide to migrating defects from Rational ClearQuest to Rational Team Concert 2.0
Mark D. Dunn, IBM
Last updated: October 14, 2009
Build basis: Rational Team Concert 2.0
This document provides step-by-step instructions on how to successfully import defects from a Rational ClearQuest database to a project area in a Rational Team Concert 2.0 server as Jazz work items (defects). It also provides some practical examples of techniques that were used to successfully migrate defects from a ClearQuest system (RATLC) to Rational Team Concert for the Rational Performance Tester development team.
Rational Team Concert 2.0 provides the capability to import ClearQuest defects into Rational Team Concert as Jazz work items. This is accomplished using the ClearQuest Importer function of the Team Concert 2.0 client. The CQ Importer is accessed from the client by selecting File > Import > Team > ClearQuest Records.
Although it is possible to import directly from the ClearQuest repository into RTC, this paper recommends using a two-step approach. This approach consists of exporting the ClearQuest defect records into a .zip file, then importing the .zip file contents into your project area in your RTC server.
An essential part of the migration is the mapping file. This XML-based file describes, field by field, how data in CQ records are imported into RTC work items. You create a mapping file template the first time that you perform an import, and you customize the mapping file according to your needs. The mapping file that you create is a copy of the sample mapping file, and it is based on both the Eclipse Way process in RTC and the ClearQuest record type from the DefectTracking schema. You modify this file repeatedly to map the data fields and user IDs (e-mail addresses) correctly before the final migration to the production RTC server.
You will want to update the RTC Process Configuration for your project area to include fields from CQ that are not defined in the default RTC Defect work item. You will probably also want to modify the Editor Presentation to place the new fields on the appropriate tabs in the RTC defect.
The most time-consuming part of the migration is the mapping of the user IDs. The users on the RTC server typically do not match the CQ users who opened, owned, or commented on defects. For this reason, request a copy of your RTC server (with its current valid user IDs and permissions) to use as a sandbox for performing test imports. This enables you to detect invalid or missing IDs in RTC and either add the new users to RTC or map invalid CQ IDs to valid RTC IDs. Because each CQ record imported into RTC creates a new work item, you must import each CQ record only one time into your production system. That is why it is important to work out all problems in the sandbox RTC server before importing into the production RTC server.
Many CQ schemas (RATLC, for instance), do not migrate attachments to RTC, because the attachments do not actually reside in CQ. Therefore, you must migrate the RATLC field named AttachmentLog to RTC. This field is a link to the attachments in RATLC. You can then move the attachment to the RTC defect.
The recommended approach is as follows:
- Import the contents of a CQ query into a .zip file using the CQ Importer. First, create an initial query to retrieve a representative set, rather than the full set, of CQ defects to be migrated. After you correct all problems with data-field mapping, use the CQ query that retrieves the full set of CQ defects to detect invalid IDs, and then map those IDs accordingly.
- Create the initial mapping file from the .zip file containing the CQ defect records. Use the CQ importer to create the initial mapping file.
- Update the RTC Process Configuration for your project area to include the new fields from CQ. You will create new Enumeration and Types and Attributes entries in the Project Configuration section of the Process Configuration for your project area. You need to create these entries to hold the new CQ information.
- Customize the mapping file to match the CQ schema and RTC Process and Project configuration you use. Perform test imports into your RTC sandbox and verify that fields and IDs are mapped correctly.
- Before you import into the production RTC server, back up your RTC repository. If an unexpected error occurs, you can always restore your server using the backup.
- Modify the Process Configuration on the production RTC server to match the new fields that you added to hold the CQ data. You can do this manually, one attribute at a time, or you can copy and paste the entire process configuration source from your test computer to the production computer (be careful that you do not overwrite previous configuration changes).
- Perform the import into the production RTC server and verify your results.
Here is a list of what you will need to perform the defect migration:
- Workstation to run all import tests.
I suggest doing all your tests from one workstation or laptop, so that you do not have to move the mapping file and .zip files from computer to computer. However, it is not mandatory to use the same computer to perform all the test imports.
- RTC 2.0 client (for defect migration and checking results).
- RTC 2.0 server (on your workstation for early tests to get the basic mapping working correctly).
- Rational ClearQuest, which includes the CQ client. The RTC CQ Importer uses the CQ client to connect to CQ.
- An RTC sandbox, which is a copy of your production RTC system.
You will need to open a RES ticket for this. This sandbox should be a copy of your production RTC system. You will need this after you have successfully mapped the RATLC data to RTC attributes. The sandbox is used primarily to determine the ID mapping errors that you need to correct.
- Administrator access in RATLC.
Submit a RAMT request to add your RATLC account to the Project Admin group. Information on how to submit the request and a link to the RAMT (Rational Account Management Tool) are provided at https://w3km.lexma.ibm.com/RESApps/Wiki.jsp?page=RATLCUserSupport.
- Access to your production RTC server.
The person performing the CQ to RTC defect migration should not have JazzAdmin privileges. This is because you do not want new user IDs to be generated in your RTC server automatically. Instead, you want to control which user IDs are valid in your RTC server.
- A CQ query that retrieves a subset of defects that are representative of the full set that you want to migrate. This query is used to work out how the CQ data maps to RTC attributes.
- A CQ query that retrieves all the defects that you want to migrate from CQ to RTC. This query is used to correct the ID mapping and to perform the migration into the production system.
- Request an RTC sandbox.
- Request RATLC administrative privileges.
- Install the RTC client.
- Install the RTC server. When you create the project areas to test with, verify that they use the same Workflow Process (for example, Scrum or Eclipse Way) that your production system uses.
- Install CQ.
- Configure the CQ client. Note that the following steps pertain to those using the RATLC database schema.
- For RATLC, download the profile for your location from: https://w3km.lexma.ibm.com/RESApps/Wiki.jsp?page=RATLCLocations
- Select Start > Programs > IBM Rational > IBM Rational ClearQuest > ClearQuest Maintenance Tool.
- From the CQ Maintenance Tool, import the profile that you downloaded by selecting File > Import Profile and pointing to the profile for your RATLC location.
- RTC server on your local workstation
Although having a RTC server on your local computer is not required, it was useful because it allowed the defect migration to begin during the wait for the RTC sandbox (the copy of the production system).
Use the RTC server installed on your local computer to determine the mapping of the RATLC data fields to RTC attributes. Since you have JazzAdmin access to your local RTC server, any invalid RATLC user IDs are automatically created in your local RTC server. Do not be concerned about ID mapping at this point. Just focus on the data field mappings.
- RTC sandbox – copy of the production RTC server
Use the RTC sandbox after you have determined the basic RATLC to RTC data mapping. In the sandbox, you do not have JazzAdmin access, and any invalid user ID that are found create an error in the error log.
The person performing the import must not have JazzAdmin privileges. This sandbox will have all the IDs currently in the production system. You should also create some additional project areas for testing. Use the copy of the production project area to do the final import test after you have the user ID mapping completed. Use the copy of the production project area to do test imports for ID mapping errors. When a RATLC record with an invalid user ID (a user ID that is not on your production RTC server) attempts to be imported, you receive an error in the error log that states that you do not have the authority to perform that operation. This is because the import is trying to create the new user ID, but you do not have JazzAdmin privileges and therefore do not have the authority to create a new user ID.
- Production RTC server
Use the production RTC server when you are ready to do the final import; that is, after you have fixed all field and ID mapping and have the mapping file correct. You also have added all the new attributes and enumerations to the Project Configuration section of the Process Configuration for your project area.
No matter how much testing you do, the final import into production will almost certainly have some errors. Review the error log to determine which RATLC defects had errors. Make a copy of the .zip file containing the RATLC records and remove all RATLC defect records except those that resulted in an error. Fix the mapping file, and rerun the import using the .zip file containing only the RATLC records which previously failed.
Remember, you must import each RATLC defect only once. A new RTC defect will be created for each RATLC defect you import, even if it was successfully imported previously. You do not want to have duplicate defects in RTC.
Detailed steps to perform the migration
A successful migration was performed by following these steps:
- Create a query in CQ that will return a representative sample of the defects that you want to migrate.
This can be done with either the CQ client or using the web interface to CQ.
- Confirm that the Query Presentation section of the CQ query contains the fields with the data you want to migrate to RTC.
You can use either the CQ client or using the web interface to CQ to select the fields retrieved by the CQ query. Remember that only data from the selected fields are placed into the CQ records contained in the .zip file, and therefore only this data is available for migration into RTC.
These fields were selected for the defect migration:
- Export the selected RATLC records into a .zip file using the CQ Importer.
The major part of the migration is done with the CQ Importer within the RTC client. To access the CQ Importer, start the RTC client and select File > Import > Team > ClearQuest Records > Next. In the following window, the RATLC user database was selected as well as the schema repository for Raleigh. You must also select the CQ query to use (click Browse to display a list of queries). Select Save as XML data and provide the .zip file name in which to save the CQ records. Click Finish. Processing this query may take a long time, depending on the number of CQ records retrieved.
Note: You receive the following error if you have not installed the CQ client and configured it to point to your RATLC database. The configuration steps are listed in the Preparation section.
- Create the initial mapping file from the CQ records in the .zip file.
First you must connect to the RTC project area that will contain the imported CQ records. To do this, access the CQ Importer again by selecting File > Import > Team > ClearQuest Records > Next.
This time, select ZIP file in the Source Location section, select Import to Project Area, select a project area in the Options section, and select Save Default As in the Custom Data Mapping section.Selecting these options provides some basic default mapping using the fields found in the CQ records within the generated .zip file.
This mapping file contains information on how to map attributes and attribute types. An attribute mapping’s sourceId (see below) refers to the element name found in the .zip file that contains the CQ defect records. For attributeType mapping, the targetId refers to the RTC attribute ID it should be mapped to. For the attribute value mapping, the sourceId refers to the value found in CQ, and the targetId refers to the RTC value to map it to. The following example shows mapping for the CQ attributes DefectSeverity, BusinessPriority, and audithistory:
<?xml version="1.0" encoding="UTF-8"?> <mapping version="1"> <attribute sourceId="DefectSeverity" targetId="com.ibm.team.workitem.attribute.severity"/> <attributeType targetId="severity"> <value sourceId="DS1" targetId="severity.literal.l5"/> <value sourceId="DS2" targetId="severity.literal.l4"/> <value sourceId="DS3" targetId="severity.literal.l3"/> <value sourceId="DS4" targetId="severity.literal.l2"/> <value sourceId="" targetId="severity.literal.l2"/> </attributeType> <attribute sourceId="BusinessPriority" targetId="com.ibm.team.workitem.attribute.priority"/> <attributeType targetId="priority"> <value sourceId="BP1" targetId="priority.literal.l11" /> <value sourceId="BP2" targetId="priority.literal.l07" /> <value sourceId="BP3" targetId="priority.literal.l02"/> <value sourceId="BP4" targetId="priority.literal.l02" /> <value sourceId="" targetId="priority.literal.l02" /> </attributeType> <attribute sourceId="audithistory" targetId="com.ibm.rational.audithistoryHTML" />
To find the RTC value to map a CQ data item to, open the project area, examine the RTC process configuration source, and find the values you need.
For a more detailed mapping discussion, see the section "Generate and Customize the Mapping File" in the "TN007: Guide for Importing Jazz Work Items from Bugzilla and Other Systems" at https://jazz.net/wiki/bin/view/Main/CQImportWizardTips and the section "Customize the Mapping File" in the "Tips on using the CQ Importer" (https://jazz.net/library/techtip/69).
- Copy the entire process configuration source from your production system project area and paste it into the process configuration for your project area on the local RTC server.
You also need to request that no process configuration modifications be made in the production system until you complete defect migration. This freeze on modifications is required so that you can copy the entire process configuration source from the system you were testing into your production system before you migrate the defects into your production system.
To copy the process configuration source:
- Log in into your production system.
- Open the project area that the defects will be migrated into. To do this, open the Team Artifacts view, right-click the project area name, and select Open.
- Click the Process Configuration Source tab.
- Click Control-A to select the entire source.
- Log in to the local RTC server that you will do test imports with.
- Open the project area in the local RTC server.
- Click the Process Configuration Source tab.
- Click Control-A to select the entire source.
- Press the Delete key to delete the source.
- Click Control-V to copy the process configuration source from the production system into the test system.
- Save the process configuration source.
- Add custom attributes to the RTC Process Configuration defect work item types.
First, test this using project areas in your local RTC server, and then in your RTC sandbox, before modifying the production RTC server. First add new enumerations, and then add new attributes, because the new attributes typically reference the new enumerations that you create.
- Open the project area: Right-click the project and select Open.
- Click Process Configuration.
- Expand Project Configuration > Configuration Data > Work Items.
- Click Enumerations, and then click Add to add new enumerations as needed. Enumerations are the equivalent to the pull-down menu choices that you see in RATLC.
The following enumerations were added during a successful migration (and used when creating the Types and Attributes entries, in the next step). The choices that you create are found in pull-down menus in your CQ system for comparable data fields:
Enumeration Examples com.ibm.rational.consumability Easy to do business com.ibm.rational.regressionfrom 6.1, 7.0, 7.0.1 (release versions) enumFoundInPlatform Windows XP, AIX 6.1 enumVersion 6.1, 7.0.1 enumBusinessPriority From prior release enumBusinessPriority BP1, BP2, BP3 com.ibm.rational.howfound Customer use com.ibm.rational.component Execute Test Schedule, Create Test enumStatus Fixed indirectly, No plan to fix
- Click Types and Attributes to add new attributes and types.
- Select Defect > com.ibm.team.workitem.workItemType to add new attributes to defects.
- Add the new custom attributes as shown below. These attributes were added during a successful RATLC to RTC migration. You must enter the Name, Type, and ID.
Note: When you create the new work item attribute in RTC that the AttachmentLog will map to, make the new attribute of type mediumHTML. This allows the value in the new attribute to be a link (rather than just a string).
Name Type ID Blocking Boolean blocking Consumability com.ibm.rational.consumability (same as Type) Found In Platform com.ibm.rational.foundinplatform (same as Type) Found In Build Small String com.ibm.rational.foundinbuild How Found com.ibm.rational.howfound (same as Type) Regression From com.ibm.rational.regressionfrom (same as Type) Regression Boolean com.ibm.rational.regression Regression Type enumRegression com.ibm.rational.regressiontype Verified In buildIds verifiedIn RATLC Component com.ibm.rational.component (same as Type) RATLC Status enumStatus com.ibm.rational.status Audit History Large HTML com.ibm.rational.audithistoryHTML RATLC Keywords Medium String com.ibm.rational.ratlckeywords RATLC Submitter Small String com.ibm.rational.ratlcsubmitter RATLC Owner Small String com.ibm.rational.status Fix In Version enumVersion com.ibm.rational.fixinversion RATLC Attachment Log Medium HTML com.ibm.rational.attachmentlogHTML
The Enumeration entries, created in step 4, and the Types and Attributes entries, created in this step, are both used to migrate RATLC defects to RTC.
- Customize the mapping file.
The following mapping was used to successfully migrate CQ (RATLC) defects to RTC. RPTRATLCMapfileSep09.xml is the customized mapping file that the Performance Tester team used.
RATLC Field RTC Field or ID Id
Imported From (automatic mapping) Headline com.ibm.team.workitem.attribute.summary
com.ibm.team.workitem.attribute.creationdate Component com.ibm.team.workitem.attribute.category Owner com.ibm.team.workitem.attribute.owner DefectSeverity com.ibm.team.workitem.attribute.severity Submitter com.ibm.team.workitem.creator FixInVersion com.ibm.rational.fixinversion (user-created attribute) CRType com.ibm.team.workitem.attribute.workitemtype State com.ibm.team.workitem.attribute.state Status com.ibm.rational.status (user-created attribute) Regression com.ibm.rational.regressiontype Description com.ibm.team.workitem.attribute.description FoundInPlatform com.ibm.rational.foundinplatform (user-created attribute) FoundInVersion com.ibm.rational.foundinversion (user-created attribute) Keywords com.ibm.rational.keywords (user-created attribute) Submitter.email com.ibm.rational.ratlcsubmitter (user-created attribute) Owner.email com.ibm.rational.ratlcowner (user-created attribute) FoundInBuild com.ibm.rational.foundinbuild (user-created attribute) Consumability com.ibm.rational.consumability (user-created attribute) HowFound com.ibm.rational.howfound (user-created attribute) RegressionPriorRelease com.ibm.rational.regressionfrom (user-created attribute)
com.ibm.rational.attachmentlogHTML Audithistory com.ibm.rational.audithistoryHTML Notes_log
Discussion (automatic mapping)
user-created attribute denotes attributes created in RTC manually to hold the new CQ data. You must verify these attributes.
- Import the RATLC records from the .zip file into your local RTC server and manually review the newly created RTC defects to verify that the field mapping is correct.
To do this, examine a sample of the newly created RTC defects and compare the RTC data to the data in the .zip file (or to the data in CQ).
To detect other types of errors, open the RTC client CQ Importer and click Window > Show View > Other > General > Error Log. This opens the error log.
- Repeat steps 6 to 8 until all the fields are mapped correctly.
Before you start the next step, double-check that all the fields are mapped correctly.
- Generate the .zip file again, this time using the CQ query that retrieves all defects to be migrated from CQ to RTC.
At this point, use the full CQ query that retrieves all the CQ records that you want to migrate to RTC. The full query results are needed in order to detect and fix all ID errors.
- Import the RATLC records from the .zip file into your RTC sandbox and detect ID errors (after copying the entire process configuration source from your local RTC server to the RTC sandbox).
Now move all your work to your RTC sandbox. You must first copy the entire process configuration source from the local RTC server (which you modified to add new attributes) into your RTC sandbox as described in step 5. This will include all the new enumerations and attributes that you added when testing your mapping file.
Make sure that you are displaying the error log so that you can see the invalid ID errors. To display the error log in the RTC CQ Importer, select Window > Show View > Other > General > Error Log.
The error log displays an error in the following form to indicate that the CQ records contain an ID (e-mail address) that is not a valid user in the RTC server.
The user “email@example.com” is not authorized to perform the operation. The “JazzAdmins” role is required to perform this operation.
To fix this, use either of the following options:
- Add the owner of the invalid address as a valid user of your RTC server.
- Map the invalid e-mail address to a valid user of your RTC server.
<attributeType targetId="contributor"> <value sourceId="Fred@example.com" targetId="firstname.lastname@example.org"/> </attributeType
You will receive errors not only from e-mail addresses of people who submitted or own the defect, but also from e-mail addresses of people who made comments in the defect. You must fix these errors and reimport those failing defects.
- Fix the IDs and verify that the mapping is correct by reviewing the newly created RTC defects in your production RTC system.
You will find one error message in the error log for each invalid address. To correct these problems,create one entry in the map file to map each invalid address, as shown below:
<attributeType targetId="contributor"> <value sourceId="Fred@example.com" targetId="email@example.com"/> <value sourceId="firstname.lastname@example.org" targetId="email@example.com"/> <value sourceId="firstname.lastname@example.org" targetId="email@example.com"/> </attributeType>
- Repeat steps 11-12 until all IDs are mapped successfully.
- Modify the Editor Presentations (still working in the RTC sandbox) to move the new information to different tabs and screen layouts, as desired.
To modify the Editor Presentations in RTC, right-click the project area and select Open. Then select Project Configuration > Configuration Data > Editor Presentations. The Editor Presentations window opens.
By default, all new custom attributes are placed in the Custom tab. You can add new tabs and move presentations to different tabs. Use Add Presentation to move an attribute to a different tab.
- Back up the production RTC server.
If an error occurs during the import, restore the production RTC system from the backup.
- Copy the process configuration source to the production system.
Follow the instructions in step 5 to copy the entire process configuration source from the system you have been testing with (your RTC sandbox), which you know is correct, to the production system.
- If necessary, run the query again and generate the .zip file that contains all the CQ defect records that you will import into the production system.
If new CQ defects have been filed (for example, too much time might have passed since you generated the .zip file), you must generate the .zip file again. The new .zip file will contain the additional CQ defects that need to be migrated.
- Import the .zip file into the production system and verify results.
If a problem occurs, manually create a new .zip file with only those CQ records that failed during the import. You do not want to import the same CQ defect twice into the production system. To create the new .zip file, copy the original .zip file to a new .zip file, and remove all the records that were successfully imported. Or conversely, you can create a new empty .zip file, and copy only the failing defects from the original .zip file to the new .zip file.
Fix the problems that occurred and import the .zip file, which contains only the CQ records that failed the import.
- Consult with your CQ team and request that the migrated defects be made obsolete in CQ.
You may want to delete the CQ records from CQ after they have been migrated to RTC.
The most common error messages are discussed in this document. For a more complete list of error messages that you might receive, follow the “Tips on Using the CQ Import Wizard” link in the “For more information section” that follows, and review the "Error Messages" section.
For more information
- Tips on using the CQ Import Wizard
- Guide for Importing Jazz Work Items from Bugzilla and Other Systems
About the author
Mark D. Dunn is a software developer and the FVT Lead for Rational Performance Tester. He can be contacted at firstname.lastname@example.org.