Guide to migrating defects from Rational ClearQuest to Rational Team Concert 2.0

Mark D. Dunn, IBM
Last updated: October 14, 2009
Build basis: Rational Team Concert 2.0

Summary

This document provides step-by-step instructions on how to successfully import defects from a Rational ClearQuest database to a project area in a Rational Team Concert 2.0 server as Jazz work items (defects). It also provides some practical examples of techniques that were used to successfully migrate defects from a ClearQuest system (RATLC) to Rational Team Concert for the Rational Performance Tester development team.

Overview

Rational Team Concert 2.0 provides the capability to import ClearQuest defects into Rational Team Concert as Jazz work items. This is accomplished using the ClearQuest Importer function of the Team Concert 2.0 client. The CQ Importer is accessed from the client by selecting File > Import > Team > ClearQuest Records.

Although it is possible to import directly from the ClearQuest repository into RTC, this paper recommends using a two-step approach. This approach consists of exporting the ClearQuest defect records into a .zip file, then importing the .zip file contents into your project area in your RTC server.

An essential part of the migration is the mapping file. This XML-based file describes, field by field, how data in CQ records are imported into RTC work items. You create a mapping file template the first time that you perform an import, and you customize the mapping file according to your needs. The mapping file that you create is a copy of the sample mapping file, and it is based on both the Eclipse Way process in RTC and the ClearQuest record type from the DefectTracking schema. You modify this file repeatedly to map the data fields and user IDs (e-mail addresses) correctly before the final migration to the production RTC server.

You will want to update the RTC Process Configuration for your project area to include fields from CQ that are not defined in the default RTC Defect work item. You will probably also want to modify the Editor Presentation to place the new fields on the appropriate tabs in the RTC defect.

The most time-consuming part of the migration is the mapping of the user IDs. The users on the RTC server typically do not match the CQ users who opened, owned, or commented on defects. For this reason, request a copy of your RTC server (with its current valid user IDs and permissions) to use as a sandbox for performing test imports. This enables you to detect invalid or missing IDs in RTC and either add the new users to RTC or map invalid CQ IDs to valid RTC IDs. Because each CQ record imported into RTC creates a new work item, you must import each CQ record only one time into your production system. That is why it is important to work out all problems in the sandbox RTC server before importing into the production RTC server.

Many CQ schemas (RATLC, for instance), do not migrate attachments to RTC, because the attachments do not actually reside in CQ. Therefore, you must migrate the RATLC field named AttachmentLog to RTC. This field is a link to the attachments in RATLC. You can then move the attachment to the RTC defect.

Recommended Approach

The recommended approach is as follows:

1. Import the contents of a CQ query into a .zip file using the CQ Importer. First, create an initial query to retrieve a representative set, rather than the full set, of CQ defects to be migrated. After you correct all problems with data-field mapping, use the CQ query that retrieves the full set of CQ defects to detect invalid IDs, and then map those IDs accordingly.
2. Create the initial mapping file from the .zip file containing the CQ defect records. Use the CQ importer to create the initial mapping file.
3. Update the RTC Process Configuration for your project area to include the new fields from CQ. You will create new Enumeration and Types and Attributes entries in the Project Configuration section of the Process Configuration for your project area. You need to create these entries to hold the new CQ information.
4. Customize the mapping file to match the CQ schema and RTC Process and Project configuration you use. Perform test imports into your RTC sandbox and verify that fields and IDs are mapped correctly.
5. Before you import into the production RTC server, back up your RTC repository. If an unexpected error occurs, you can always restore your server using the backup.
6. Modify the Process Configuration on the production RTC server to match the new fields that you added to hold the CQ data. You can do this manually, one attribute at a time, or you can copy and paste the entire process configuration source from your test computer to the production computer (be careful that you do not overwrite previous configuration changes).
7. Perform the import into the production RTC server and verify your results.

Getting Started

Here is a list of what you will need to perform the defect migration:

1. Workstation to run all import tests.

   I suggest doing all your tests from one workstation or laptop, so that you do not have to move the mapping file and .zip files from computer to computer. However, it is not mandatory to use the same computer to perform all the test imports.

2. RTC 2.0 client (for defect migration and checking results).

   https://jazz.net/downloads/rational-team-concert/releases/2.0/RTC-Client-2.0-Win.zip

3. RTC 2.0 server (on your workstation for early tests to get the basic mapping working correctly).

   https://jazz.net/downloads/rational-team-concert/releases/2.0/RTC-Standard-Server-2.0-Trial-Win32.zip

4. Rational ClearQuest, which includes the CQ client. The RTC CQ Importer uses the CQ client to connect to CQ.

   https://w3km.lexma.ibm.com/RESApps/Wiki.jsp?page=DownloadCQ701

5. An RTC sandbox, which is a copy of your production RTC system.

   You will need to open a RES ticket for this. This sandbox should be a copy of your production RTC system. You will need this after you have successfully mapped the RATLC data to RTC attributes. The sandbox is used primarily to determine the ID mapping errors that you need to correct.

6. Administrator access in RATLC.

   Submit a RAMT request to add your RATLC account to the Project Admin group. Information on how to submit the request and a link to the RAMT (Rational Account Management Tool) are provided at https://w3km.lexma.ibm.com/RESApps/Wiki.jsp?page=RATLCUserSupport.

7. Access to your production RTC server.

   The person performing the CQ to RTC defect migration should not have JazzAdmin privileges. This is because you do not want new user IDs to be generated in your RTC server automatically. Instead, you want to control which user IDs are valid in your RTC server.

8. A CQ query that retrieves a subset of defects that are representative of the full set that you want to migrate. This query is used to work out how the CQ data maps to RTC attributes.
9. A CQ query that retrieves all the defects that you want to migrate from CQ to RTC. This query is used to correct the ID mapping and to perform the migration into the production system.

Preparation

1. Request an RTC sandbox.
2. Request RATLC administrative privileges.
3. Install the RTC client.
4. Install the RTC server. When you create the project areas to test with, verify that they use the same Workflow Process (for example, Scrum or Eclipse Way) that your production system uses.
5. Install CQ.
6. Configure the CQ client. Note that the following steps pertain to those using the RATLC database schema.
   1. For RATLC, download the profile for your location from: https://w3km.lexma.ibm.com/RESApps/Wiki.jsp?page=RATLCLocations
   2. Select Start > Programs > IBM Rational > IBM Rational ClearQuest > ClearQuest Maintenance Tool.
   3. From the CQ Maintenance Tool, import the profile that you downloaded by selecting File > Import Profile and pointing to the profile for your RATLC location.

Suggested RTC servers to use in defect migration process

1. RTC server on your local workstation

   Although having a RTC server on your local computer is not required, it was useful because it allowed the defect migration to begin during the wait for the RTC sandbox (the copy of the production system).

   Use the RTC server installed on your local computer to determine the mapping of the RATLC data fields to RTC attributes. Since you have JazzAdmin access to your local RTC server, any invalid RATLC user IDs are automatically created in your local RTC server. Do not be concerned about ID mapping at this point. Just focus on the data field mappings.

2. RTC sandbox – copy of the production RTC server

   Use the RTC sandbox after you have determined the basic RATLC to RTC data mapping. In the sandbox, you do not have JazzAdmin access, and any invalid user ID that are found create an error in the error log.

   The person performing the import must not have JazzAdmin privileges. This sandbox will have all the IDs currently in the production system. You should also create some additional project areas for testing. Use the copy of the production project area to do the final import test after you have the user ID mapping completed. Use the copy of the production project area to do test imports for ID mapping errors. When a RATLC record with an invalid user ID (a user ID that is not on your production RTC server) attempts to be imported, you receive an error in the error log that states that you do not have the authority to perform that operation. This is because the import is trying to create the new user ID, but you do not have JazzAdmin privileges and therefore do not have the authority to create a new user ID.

3. Production RTC server

   Use the production RTC server when you are ready to do the final import; that is, after you have fixed all field and ID mapping and have the mapping file correct. You also have added all the new attributes and enumerations to the Project Configuration section of the Process Configuration for your project area.

   No matter how much testing you do, the final import into production will almost certainly have some errors. Review the error log to determine which RATLC defects had errors. Make a copy of the .zip file containing the RATLC records and remove all RATLC defect records except those that resulted in an error. Fix the mapping file, and rerun the import using the .zip file containing only the RATLC records which previously failed.

   Remember, you must import each RATLC defect only once. A new RTC defect will be created for each RATLC defect you import, even if it was successfully imported previously. You do not want to have duplicate defects in RTC.

Detailed steps to perform the migration

A successful migration was performed by following these steps:

1. Create a query in CQ that will return a representative sample of the defects that you want to migrate.

   This can be done with either the CQ client or using the web interface to CQ.

2. Confirm that the Query Presentation section of the CQ query contains the fields with the data you want to migrate to RTC.

   You can use either the CQ client or using the web interface to CQ to select the fields retrieved by the CQ query. Remember that only data from the selected fields are placed into the CQ records contained in the .zip file, and therefore only this data is available for migration into RTC.

   These fields were selected for the defect migration:

   • FoundInPlatform
   • Headline
   • FoundInVersion
   • SubmitDate
   • Keywords
   • Component
   • Submitter.email
   • Owner
   • Owner.email
   • DefectSeverity
   • FoundInBuild
   • Submitter
   • Notes_Log
   • FixInVersion
   • Consumability
   • CRType
   • HowFound
   • State
   • RegressionPriorRelease
   • Status
   • audithistory
   • Regression
   • AttachmentLog
   • Description
3. Export the selected RATLC records into a .zip file using the CQ Importer.

   The major part of the migration is done with the CQ Importer within the RTC client. To access the CQ Importer, start the RTC client and select File > Import > Team > ClearQuest Records > Next. In the following window, the RATLC user database was selected as well as the schema repository for Raleigh. You must also select the CQ query to use (click Browse to display a list of queries). Select Save as XML data and provide the .zip file name in which to save the CQ records. Click Finish. Processing this query may take a long time, depending on the number of CQ records retrieved.
   

   
   Note: You receive the following error if you have not installed the CQ client and configured it to point to your RATLC database. The configuration steps are listed in the Preparation section.
   

4. Create the initial mapping file from the CQ records in the .zip file.

   First you must connect to the RTC project area that will contain the imported CQ records. To do this, access the CQ Importer again by selecting File > Import > Team > ClearQuest Records > Next.

   This time, select ZIP file in the Source Location section, select Import to Project Area, select a project area in the Options section, and select Save Default As in the Custom Data Mapping section.

   Selecting these options provides some basic default mapping using the fields found in the CQ records within the generated .zip file.

   This mapping file contains information on how to map attributes and attribute types. An attribute mapping’s sourceId (see below) refers to the element name found in the .zip file that contains the CQ defect records. For attributeType mapping, the targetId refers to the RTC attribute ID it should be mapped to. For the attribute value mapping, the sourceId refers to the value found in CQ, and the targetId refers to the RTC value to map it to. The following example shows mapping for the CQ attributes DefectSeverity, BusinessPriority, and audithistory:

           <?xml version="1.0" encoding="UTF-8"?>        <mapping version="1">        <attribute sourceId="DefectSeverity"             targetId="com.ibm.team.workitem.attribute.severity"/>         <attributeType targetId="severity">           <value sourceId="DS1" targetId="severity.literal.l5"/>           <value sourceId="DS2" targetId="severity.literal.l4"/>           <value sourceId="DS3" targetId="severity.literal.l3"/>           <value sourceId="DS4" targetId="severity.literal.l2"/>           <value sourceId="" targetId="severity.literal.l2"/>         </attributeType>         <attribute sourceId="BusinessPriority"            targetId="com.ibm.team.workitem.attribute.priority"/>                   <attributeType targetId="priority">          <value sourceId="BP1" targetId="priority.literal.l11" />          <value sourceId="BP2" targetId="priority.literal.l07" />          <value sourceId="BP3" targetId="priority.literal.l02"/>          <value sourceId="BP4" targetId="priority.literal.l02" />          <value sourceId="" targetId="priority.literal.l02" />         </attributeType>        <attribute sourceId="audithistory"           targetId="com.ibm.rational.audithistoryHTML" />

   To find the RTC value to map a CQ data item to, open the project area, examine the RTC process configuration source, and find the values you need.

   For a more detailed mapping discussion, see the section "Generate and Customize the Mapping File" in the "TN007: Guide for Importing Jazz Work Items from Bugzilla and Other Systems" at https://jazz.net/wiki/bin/view/Main/CQImportWizardTips and the section "Customize the Mapping File" in the "Tips on using the CQ Importer" (https://jazz.net/library/techtip/69).

5. Copy the entire process configuration source from your production system project area and paste it into the process configuration for your project area on the local RTC server.

   You also need to request that no process configuration modifications be made in the production system until you complete defect migration. This freeze on modifications is required so that you can copy the entire process configuration source from the system you were testing into your production system before you migrate the defects into your production system.

   To copy the process configuration source:

   1. Log in into your production system.
   2. Open the project area that the defects will be migrated into. To do this, open the Team Artifacts view, right-click the project area name, and select Open.
   3. Click the Process Configuration Source tab.
   4. Click Control-A to select the entire source.
   5. Log in to the local RTC server that you will do test imports with.
   6. Open the project area in the local RTC server.
   7. Click the Process Configuration Source tab.
   8. Click Control-A to select the entire source.
   9. Press the Delete key to delete the source.
   10. Click Control-V to copy the process configuration source from the production system into the test system.
   11. Save the process configuration source.
6. Add custom attributes to the RTC Process Configuration defect work item types.

   First, test this using project areas in your local RTC server, and then in your RTC sandbox, before modifying the production RTC server. First add new enumerations, and then add new attributes, because the new attributes typically reference the new enumerations that you create.

   1. Open the project area: Right-click the project and select Open.
   2. Click Process Configuration.
   3. Expand Project Configuration > Configuration Data > Work Items.
   4. Click Enumerations, and then click Add to add new enumerations as needed. Enumerations are the equivalent to the pull-down menu choices that you see in RATLC.

      The following enumerations were added during a successful migration (and used when creating the Types and Attributes entries, in the next step). The choices that you create are found in pull-down menus in your CQ system for comparable data fields:

      Enumeration	Examples
      com.ibm.rational.consumability	Easy to do business
      com.ibm.rational.regressionfrom	6.1, 7.0, 7.0.1 (release versions)
      enumFoundInPlatform	Windows XP, AIX 6.1
      enumVersion	6.1, 7.0.1
      enumBusinessPriority	From prior release
      enumBusinessPriority	BP1, BP2, BP3
      com.ibm.rational.howfound	Customer use
      com.ibm.rational.component	Execute Test Schedule, Create Test
      enumStatus	Fixed indirectly, No plan to fix

   5. Click Types and Attributes to add new attributes and types.
   6. Select Defect > com.ibm.team.workitem.workItemType to add new attributes to defects.
   7. Add the new custom attributes as shown below. These attributes were added during a successful RATLC to RTC migration. You must enter the Name, Type, and ID.

      Note: When you create the new work item attribute in RTC that the AttachmentLog will map to, make the new attribute of type mediumHTML. This allows the value in the new attribute to be a link (rather than just a string).

      Name	Type	ID
      Blocking	Boolean	blocking
      Consumability	com.ibm.rational.consumability	(same as Type)
      Found In Platform	com.ibm.rational.foundinplatform	(same as Type)
      Found In Build	Small String	com.ibm.rational.foundinbuild
      How Found	com.ibm.rational.howfound	(same as Type)
      Regression From	com.ibm.rational.regressionfrom	(same as Type)
      Regression	Boolean	com.ibm.rational.regression
      Regression Type	enumRegression	com.ibm.rational.regressiontype
      Verified In	buildIds	verifiedIn
      RATLC Component	com.ibm.rational.component	(same as Type)
      RATLC Status	enumStatus	com.ibm.rational.status
      Audit History	Large HTML	com.ibm.rational.audithistoryHTML
      RATLC Keywords	Medium String	com.ibm.rational.ratlckeywords
      RATLC Submitter	Small String	com.ibm.rational.ratlcsubmitter
      RATLC Owner	Small String	com.ibm.rational.status
      Fix In Version	enumVersion	com.ibm.rational.fixinversion
      RATLC Attachment Log	Medium HTML	com.ibm.rational.attachmentlogHTML

      The Enumeration entries, created in step 4, and the Types and Attributes entries, created in this step, are both used to migrate RATLC defects to RTC.

7. Customize the mapping file.

   The following mapping was used to successfully migrate CQ (RATLC) defects to RTC. RPTRATLCMapfileSep09.xml is the customized mapping file that the Performance Tester team used.

   RATLC Field	RTC Field or ID
   Id	Imported From (automatic mapping)
   Headline	com.ibm.team.workitem.attribute.summary
   SubmitDate	com.ibm.team.workitem.attribute.creationdate
   Component	com.ibm.team.workitem.attribute.category
   Owner	com.ibm.team.workitem.attribute.owner
   DefectSeverity	com.ibm.team.workitem.attribute.severity
   Submitter	com.ibm.team.workitem.creator
   FixInVersion	com.ibm.rational.fixinversion (user-created attribute)
   CRType	com.ibm.team.workitem.attribute.workitemtype
   State	com.ibm.team.workitem.attribute.state
   Status	com.ibm.rational.status (user-created attribute)
   Regression	com.ibm.rational.regressiontype
   Description	com.ibm.team.workitem.attribute.description
   FoundInPlatform	com.ibm.rational.foundinplatform (user-created attribute)
   FoundInVersion	com.ibm.rational.foundinversion (user-created attribute)
   Keywords	com.ibm.rational.keywords (user-created attribute)
   Submitter.email	com.ibm.rational.ratlcsubmitter (user-created attribute)
   Owner.email	com.ibm.rational.ratlcowner (user-created attribute)
   FoundInBuild	com.ibm.rational.foundinbuild (user-created attribute)
   Consumability	com.ibm.rational.consumability (user-created attribute)
   HowFound	com.ibm.rational.howfound (user-created attribute)
   RegressionPriorRelease	com.ibm.rational.regressionfrom (user-created attribute)
   AttachmentLog	com.ibm.rational.attachmentlogHTML
   Audithistory	com.ibm.rational.audithistoryHTML
   Notes_log	Discussion (automatic mapping)

   automatic mapping indicates that the CQ importer mapped these attributes automatically. You do not have to change these.

   user-created attribute denotes attributes created in RTC manually to hold the new CQ data. You must verify these attributes.

8. Import the RATLC records from the .zip file into your local RTC server and manually review the newly created RTC defects to verify that the field mapping is correct.

   To do this, examine a sample of the newly created RTC defects and compare the RTC data to the data in the .zip file (or to the data in CQ).

   To detect other types of errors, open the RTC client CQ Importer and click Window > Show View > Other > General > Error Log. This opens the error log.

9. Repeat steps 6 to 8 until all the fields are mapped correctly.

   Before you start the next step, double-check that all the fields are mapped correctly.

10. Generate the .zip file again, this time using the CQ query that retrieves all defects to be migrated from CQ to RTC.

    At this point, use the full CQ query that retrieves all the CQ records that you want to migrate to RTC. The full query results are needed in order to detect and fix all ID errors.

11. Import the RATLC records from the .zip file into your RTC sandbox and detect ID errors (after copying the entire process configuration source from your local RTC server to the RTC sandbox).

    Now move all your work to your RTC sandbox. You must first copy the entire process configuration source from the local RTC server (which you modified to add new attributes) into your RTC sandbox as described in step 5. This will include all the new enumerations and attributes that you added when testing your mapping file.

    Make sure that you are displaying the error log so that you can see the invalid ID errors. To display the error log in the RTC CQ Importer, select Window > Show View > Other > General > Error Log.

    The error log displays an error in the following form to indicate that the CQ records contain an ID (e-mail address) that is not a valid user in the RTC server.

    The user “fred@example.com” is not authorized to perform the operation. The “JazzAdmins” role is required to perform this operation.

    To fix this, use either of the following options:

    • Add the owner of the invalid address as a valid user of your RTC server.
    • Map the invalid e-mail address to a valid user of your RTC server.
    The following example shows how to map an invalid address to a valid one (this assumes “mdunn@us.ibm.com” is a valid address in RTC):

            <attributeType targetId="contributor">         <value sourceId="Fred@example.com" targetId="mdunn@us.ibm.com"/>         </attributeType        

    You will receive errors not only from e-mail addresses of people who submitted or own the defect, but also from e-mail addresses of people who made comments in the defect. You must fix these errors and reimport those failing defects.

12. Fix the IDs and verify that the mapping is correct by reviewing the newly created RTC defects in your production RTC system.

    You will find one error message in the error log for each invalid address. To correct these problems,create one entry in the map file to map each invalid address, as shown below:

          <attributeType targetId="contributor">      <value sourceId="Fred@example.com" targetId="mdunn@us.ibm.com"/>      <value sourceId="betty@example.com" targetId="mdunn@us.ibm.com"/>      <value sourceId="dino@example.com" targetId="mdunn@us.ibm.com"/>      </attributeType>

13. Repeat steps 11-12 until all IDs are mapped successfully.
14. Modify the Editor Presentations (still working in the RTC sandbox) to move the new information to different tabs and screen layouts, as desired.

    To modify the Editor Presentations in RTC, right-click the project area and select Open. Then select Project Configuration > Configuration Data > Editor Presentations. The Editor Presentations window opens.

    

    By default, all new custom attributes are placed in the Custom tab. You can add new tabs and move presentations to different tabs. Use Add Presentation to move an attribute to a different tab.

15. Back up the production RTC server.

    If an error occurs during the import, restore the production RTC system from the backup.

16. Copy the process configuration source to the production system.

    Follow the instructions in step 5 to copy the entire process configuration source from the system you have been testing with (your RTC sandbox), which you know is correct, to the production system.

17. If necessary, run the query again and generate the .zip file that contains all the CQ defect records that you will import into the production system.

    If new CQ defects have been filed (for example, too much time might have passed since you generated the .zip file), you must generate the .zip file again. The new .zip file will contain the additional CQ defects that need to be migrated.

18. Import the .zip file into the production system and verify results.

    If a problem occurs, manually create a new .zip file with only those CQ records that failed during the import. You do not want to import the same CQ defect twice into the production system. To create the new .zip file, copy the original .zip file to a new .zip file, and remove all the records that were successfully imported. Or conversely, you can create a new empty .zip file, and copy only the failing defects from the original .zip file to the new .zip file.

    Fix the problems that occurred and import the .zip file, which contains only the CQ records that failed the import.

19. Consult with your CQ team and request that the migrated defects be made obsolete in CQ.

    You may want to delete the CQ records from CQ after they have been migrated to RTC.

Error messages

The most common error messages are discussed in this document. For a more complete list of error messages that you might receive, follow the “Tips on Using the CQ Import Wizard” link in the “For more information section” that follows, and review the "Error Messages" section.

For more information

• Tips on using the CQ Import Wizard
• Guide for Importing Jazz Work Items from Bugzilla and Other Systems

About the author

Mark D. Dunn is a software developer and the FVT Lead for Rational Performance Tester. He can be contacted at mdunn@us.ibm.com.