Migrating large data sets into RTC
I am working with a customer that has a large amount of data to migrate from another tool into RTC/RQM. Much of the data can be exported into String type data, into csv files. However a large portion of the data is represented as CLOB (attachments, comments, and other multi-line fields). I am thinking of loading the "easy" data in first. And then using data base tools to load in the large attachment data and other CLOB data, then adjusting the links table to point to the relevant entries. This is a strategy that we have used many times with ClearQuest. With RTC, I've yet to do this, so I am not as familiar with the table structure relating the attachments, links and other internal representation. Any comments on this approach? I am also interested in any performance related data you may have achieved in migrating or loading large data sets into RTC's work item module.
4 answers
I am working with a customer that has a large amount of data to migrate from another tool into RTC/RQM. Much of the data can be exported into String type data, into csv files. However a large portion of the data is represented as CLOB (attachments, comments, and other multi-line fields). I am thinking of loading the "easy" data in first. And then using data base tools to load in the large attachment data and other CLOB data, then adjusting the links table to point to the relevant entries. This is a strategy that we have used many times with ClearQuest. With RTC, I've yet to do this, so I am not as familiar with the table structure relating the attachments, links and other internal representation. Any comments on this approach? I am also interested in any performance related data you may have achieved in migrating or loading large data sets into RTC's work item module.
Hi Dave,
I have always tried to avoid manipulating the database directly.
I have created an importer myself that was able to import work item data and upgrade existing work items in the past, when the CSV import was not available. I used the plain java API for that.
Dependent on the format of the input file(s) such a solution would also be able to upload multi line descriptions. It could also be used in conjunction to the existing CSV import. It could run multiple passes for efficiency and to create relationships between artifacts in a pass after the elements have been imported.
If the exported attachments and the file name data for these attachments where available in the input data it would be possible to upload that data. There might be example implementations for this. I have example code for most of the pieces.
Hi Dave - do you have knowledge of home grown / custom tool to export work items from a 2.0.0.2 RTC instance (including attachments - thats the important part), to a separate server running 3.0.1?
Thanks
Thanks
Comments
Our group has recently created a utility to export/move work items from old repositories (2.x) into new CCM repositories (3.x). We also retain the history, attachments, everything. The customer requirements were "make it like it was created in the 3.x repo". So, we invested a lot of time in putting this together. If you or your customer is interested in also using this utility, please contact our group. You can contact one of the following people to get started:
doanminh@us.ibm.com hollisc@ca.ibm.com