Options for converting a very large existing environment (SVN & CVS) to RTC
- about 100 SVN respositories (with a format labeled as "uncommon" by the SVN2RTC conversion documentation)
- about 80 CVS repositories
- 20-40 repositories in a mix of other formats (Source Safe, Mercurial, Git)
- about half of our repositories are large: 10-15 years of history, with 10+ branches out of which 5-6 are actually maintenance branches that get developed on (or at least fixes get backported on them and we get nightly builds on them)
And I need to convert all of the above to RTC. I would consider the "mix of other formats" on a case-by-case basis, however I would expect to have SVN and CVS converted in an uniform matter. The CVS conversion it seems should be done via an intermediate cvs2svn conversion, to preserve branches and tags.
In any case, I tried to gather as much information about this as possible. I installed RTC in a test environment on a VM (with DB2 and everything, due to the sheer size of our repositories), and gathered also the following on-line documentation:
https://jazz.net/forum/questions/61525/svn-and-cvs-migration-to-rtc - asks about CSV/SVN Migration on windows, however it has some good links. Specifically:
- Main migration description: https://jazz.net/library/article/650
- Migration planning guide: https://jazz.net/library/article/70
- Concept comparison between SVN and RTC: https://jazz.net/library/article/639
- Easing into Jazz Source Control: https://jazz.net/library/article/539
Supported import layouts:
- http://pic.dhe.ibm.com/infocenter/rtc/v2r0m0/topic/com.ibm.team.scm.svn.doc/topics/c_svn.html
Is command-line supported (with the answer being: NO, not supported):
- https://jazz.net/forum/questions/25604/command-line-import-from-svn
Points that are giving me trouble:
a) "Repository organizations where the trunk folder itself maps to an Eclipse project are uncommon and are not supported by the importer:"
We have about 100 of these "uncommon" projects, and our tried-and-tested cvs-to-svn scripts would convert any cvs repository to this format, resulting in 200 of the uncommon layouts. My best solution so far was to update the svn dump by either using something like https://github.com/jwiegley/svndumptool or script one myself based on the documentation at http://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt
b) "the SVN importer only imports a single branch at a time"
We have maintenance requirements for about 5-6 branches for most of the projects, and in general I think nobody would like to lose history or have "new code on RTC, old code in svn/cvs/something else". This makes it so that not only do we have 100+ projects, but each project will need 6-12 branches imported as well.
c) No scripting support, only eclipse GUI.
GUI-based, manual conversion seems highly impractical due to the sheer volume of data. Humans make mistakes when repeating the same task over and over again.
What would be some viable options for us doing the conversion?
I am looking at importing 100s of branches it seems, with quite a few of them being in the 2-5 hour import range according to the planning guide. I considered scripting eclipse, however http://wiki.eclipse.org/E4/Scripting shows no progress, and even if Eclipse Monkey or some of it's forks work, this looks like an awful hack. Is there anything I am missing?
One answer
Comments
Does this still have to be inside eclipse or can it be a separate application?
Yes, the current state of the framework depends on the Eclipse UI. The benefit is you could write your own importer that would run the import without any other interaction from a user or handle any deviations from SVN conventions.
I forgot to mention that you should be able to import your "uncommon" projects with the Eclipse client SVN importer (ie. import the directory you consider trunk). You'll have to make some changes after the import to get the directory structure that you want. It will probably show up in RTC as /trunk/your_project_files and you would want to rename trunk to whatever your project is called.
Do you have documentation about the classes used in the Eclipse UI and how to extend a custom importer? Without it, the eclipse integration is not a benefit at all - quite the opposite - writing a custom eclipse importer seems significantly more complex than any other cross-scm import that I have seen.
From the looks of it, I could probably create a custom TAR file, as documented in https://jazz.net/wiki/bin/view/Main/SCMChangeSetArchive (altough that seems reasonably painful as well ... and insanely large as I have to put a full copy of all changed files).
If I create such a tar file, how would such a thing get imported into RTC? Since it is our custom TAR file, I can fix up any paths in any way we wish. However I am unsure about how users get mapped and also about how baselines get created.
Unfortunately, that is all the documentation available at this time. It's not going to be an easy task. It's also not easy to have your entire import automated. The SVN importer handles a typical SVN repo very well but any deviations from standard will require manual intervention to ensure it is imported correctly or very specific code to handle the case.
The TAR file is the archive that has the content and the change sets that are to be created to import the content. It's the data after converting from the other source control into something that makes sense to RTC. I think a TAR entry is created to describe the baselines to create. I didn't write the importer so I may be wrong. I don't know if you can just give this TAR file to the SVN importer and have it work. It might require you to rewrite other parts of the importer to handle the issue that you want to be able to import trunk and multiple branches at the same time.
How can I rewrite parts of the importer? Do I have source access to it?
You would implement the parts listed in the link in my answer. That's about as much as I know about the import framework.
The SDK might be of help here https://jazz.net/wiki/bin/view/Main/RtcSdk20. I'm not sure if it provides what you need though. You can get it from the all downloads section of a release and it seems to imply that source is included.
Any chance to get some information about how to mark a baseline during the tar file import? I could not find this documented in the Tar file format links nor does the sample tar contain such a thing.
Alternatively, is there a way to make a baseline "after the fact", using lscm? Something like "crate baseline named x @revision y".
I may have misled you into writing your own importer. It's poorly documented and I would be of little help. If you wanted to go this route, it would be a lot of reverse engineering but the option is there if you didn't want to run through the Eclipse SVN importer for each branch.
You might be better off scripting the import using lscm if you are comfortable using its commands to perform the import. You can certainly create baselines wherever you need it with lscm. You can't specify a baseline at a certain revision because RTC works on component configurations. So you would have all the change sets (a change set would be the equivalent of a SVN revision) in your component that you want to baseline and then create it. The content doesn't have to be loaded for this either.
So using lscm, assuming I have something like:
╰─$ lscm history
Password (alitvin @ https://testserver:9443/ccm/):
Change sets:
(1008) ---$ Andrei Litvin "foo" 16-Apr-2013 02:02 PM
(1000) ---$ Andrei Litvin "bar" 18-Oct-2007 01:09 PM
(1001) ---$ Andrei Litvin "baz" 16-Oct-2007 01:56 PM
(1002) ---$ Andrei Litvin "bat" 16-Oct-2007 01:42 PM
(1003) ---$ Andrei Litvin "blah" 16-Oct-2007 01:38 PM
(1004) ---$ Andrei Litvin "first" 16-Apr-2013 11:47 AM
Can I create a baseline at some specific place, like 1002 (either by time or by changeset number)? If I create a tar-based import, I will have all revisions however none of the tags created. I did try using lscm to do this but had no success.
Seems somewhat related to the recent question of https://jazz.net/forum/questions/109893/get-back-to-a-specific-changeset
You would discard or suspend the change sets above 1002 to get your component's configuration to the state where you want the baseline. Then you create the baseline.
Is there an easy command to suspend all the changesets after a particular version? We tag every nightly build, so the number of times I need to do this for each import is in the 100s (actually 1000s for some older branches).
What I have is the history for 10+ years, with a requirement to tag a specific version that occured a couple of years ago.
Comments
Andrei Litvin
Apr 03 '13, 8:57 a.m.Some info I forgot: we are working in a mix of windows and unix environment, however the majority of work is done in unix. We are very comfortable to script anything if at all possible (shell, perl, ruby, python ... anything that works).
Krzysztof Kaźmierczyk
Apr 03 '13, 9:35 a.m.What else you can consider is importing to git repositories and use git adapter and git as scm.
1 vote
Andrei Litvin
Apr 03 '13, 9:55 a.m.Git is low priority right now - it is 1 or 2 repositories out of 200+.
Tim Mok
JAZZ DEVELOPER Apr 03 '13, 11:49 a.m.I think Krzysztof's suggestion is to import everything into git and use it with the git adapter instead of importing into RTC.
Andrei Litvin
Apr 03 '13, 11:56 a.m.For that use case, that only ads complexity to us: we don't plan in the first phase to use RTC bug-reporting capabilities, as we have a highly customized Clearquest deployment and it would be too complex to update that one as well.
So we are looking to use RTC for our source code repository, consolidating all SVN,CVS and others into one. If we go to the git route, RTC would only add a complexity layer over git, so the proposal would amount for us to "don't convert to RTC, just use git". Management won't like it :-)
Or can this be a transitional phase? Like old history via git adapter and new history in RTC? I have not looked at the git adapter at all since it only seemed to me to link commits to bug reports (and we already have that in our environment via SVN/CVS/Git commit hooks and Clearquest).
Geoffrey Clemm
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Apr 03 '13, 7:00 p.m.You are correct that the git adapter just links commits to bug reports, so a conversion to GIT would not help in a transition to RTC-SCM.