Options for converting a very large existing environment (SVN & CVS) to RTC
We are looking at performing a large migration from existing source control (mostly SVN and CVS ... and we can automate CVS to SVN as an intermediate step) to Rational Team Concert. However the size of our environment is giving me trouble, since I cannot see any way of automating things. In general, our environment looks like this:
- about 100 SVN respositories (with a format labeled as "uncommon" by the SVN2RTC conversion documentation) - about 80 CVS repositories - 20-40 repositories in a mix of other formats (Source Safe, Mercurial, Git) - about half of our repositories are large: 10-15 years of history, with 10+ branches out of which 5-6 are actually maintenance branches that get developed on (or at least fixes get backported on them and we get nightly builds on them) And I need to convert all of the above to RTC. I would consider the "mix of other formats" on a case-by-case basis, however I would expect to have SVN and CVS converted in an uniform matter. The CVS conversion it seems should be done via an intermediate cvs2svn conversion, to preserve branches and tags. In any case, I tried to gather as much information about this as possible. I installed RTC in a test environment on a VM (with DB2 and everything, due to the sheer size of our repositories), and gathered also the following on-line documentation: https://jazz.net/forum/questions/61525/svn-and-cvs-migration-to-rtc - asks about CSV/SVN Migration on windows, however it has some good links. Specifically: - Main migration description: https://jazz.net/library/article/650 - Migration planning guide: https://jazz.net/library/article/70 - Concept comparison between SVN and RTC: https://jazz.net/library/article/639 - Easing into Jazz Source Control: https://jazz.net/library/article/539 Supported import layouts: - http://pic.dhe.ibm.com/infocenter/rtc/v2r0m0/topic/com.ibm.team.scm.svn.doc/topics/c_svn.html Is command-line supported (with the answer being: NO, not supported): - https://jazz.net/forum/questions/25604/command-line-import-from-svn Points that are giving me trouble: a) "Repository organizations where the trunk folder itself maps to an Eclipse project are uncommon and are not supported by the importer:" We have about 100 of these "uncommon" projects, and our tried-and-tested cvs-to-svn scripts would convert any cvs repository to this format, resulting in 200 of the uncommon layouts. My best solution so far was to update the svn dump by either using something like https://github.com/jwiegley/svndumptool or script one myself based on the documentation at http://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt b) "the SVN importer only imports a single branch at a time" We have maintenance requirements for about 5-6 branches for most of the projects, and in general I think nobody would like to lose history or have "new code on RTC, old code in svn/cvs/something else". This makes it so that not only do we have 100+ projects, but each project will need 6-12 branches imported as well. c) No scripting support, only eclipse GUI. GUI-based, manual conversion seems highly impractical due to the sheer volume of data. Humans make mistakes when repeating the same task over and over again. What would be some viable options for us doing the conversion? I am looking at importing 100s of branches it seems, with quite a few of them being in the 2-5 hour import range according to the planning guide. I considered scripting eclipse, however http://wiki.eclipse.org/E4/Scripting shows no progress, and even if Eclipse Monkey or some of it's forks work, this looks like an awful hack. Is there anything I am missing?
showing 5 of 6
show 1 more comments
|
One answer
You could write your own importer by extending the import framework and you would have to do it for each of the other repo formats (git, Mercurial, etc.) if you can't convert them to SVN. The documentation for this is sparse (https://jazz.net/wiki/bin/view/Main/SCMImportFromOther) and this would be quite a bit of work if your SVN repo structure deviates from standard (ie. trunk, branches, tags). However, it would give you better control on making the import automated and you can handle the specific way you use SVN.
Comments Does this still have to be inside eclipse or can it be a separate application?
Yes, the current state of the framework depends on the Eclipse UI. The benefit is you could write your own importer that would run the import without any other interaction from a user or handle any deviations from SVN conventions.
Do you have documentation about the classes used in the Eclipse UI and how to extend a custom importer? Without it, the eclipse integration is not a benefit at all - quite the opposite - writing a custom eclipse importer seems significantly more complex than any other cross-scm import that I have seen.
From the looks of it, I could probably create a custom TAR file, as documented in https://jazz.net/wiki/bin/view/Main/SCMChangeSetArchive (altough that seems reasonably painful as well ... and insanely large as I have to put a full copy of all changed files).
Unfortunately, that is all the documentation available at this time. It's not going to be an easy task. It's also not easy to have your entire import automated. The SVN importer handles a typical SVN repo very well but any deviations from standard will require manual intervention to ensure it is imported correctly or very specific code to handle the case.
How can I rewrite parts of the importer? Do I have source access to it?
You would implement the parts listed in the link in my answer. That's about as much as I know about the import framework.
Any chance to get some information about how to mark a baseline during the tar file import? I could not find this documented in the Tar file format links nor does the sample tar contain such a thing.
I may have misled you into writing your own importer. It's poorly documented and I would be of little help. If you wanted to go this route, it would be a lot of reverse engineering but the option is there if you didn't want to run through the Eclipse SVN importer for each branch.
So using lscm, assuming I have something like:
You would discard or suspend the change sets above 1002 to get your component's configuration to the state where you want the baseline. Then you create the baseline.
Is there an easy command to suspend all the changesets after a particular version? We tag every nightly build, so the number of times I need to do this for each import is in the 100s (actually 1000s for some older branches).
showing 5 of 12
show 7 more comments
|
Your answer
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.
Comments
Some info I forgot: we are working in a mix of windows and unix environment, however the majority of work is done in unix. We are very comfortable to script anything if at all possible (shell, perl, ruby, python ... anything that works).
What else you can consider is importing to git repositories and use git adapter and git as scm.
Git is low priority right now - it is 1 or 2 repositories out of 200+.
I think Krzysztof's suggestion is to import everything into git and use it with the git adapter instead of importing into RTC.
For that use case, that only ads complexity to us: we don't plan in the first phase to use RTC bug-reporting capabilities, as we have a highly customized Clearquest deployment and it would be too complex to update that one as well.
So we are looking to use RTC for our source code repository, consolidating all SVN,CVS and others into one. If we go to the git route, RTC would only add a complexity layer over git, so the proposal would amount for us to "don't convert to RTC, just use git". Management won't like it :-)
Or can this be a transitional phase? Like old history via git adapter and new history in RTC? I have not looked at the git adapter at all since it only seemed to me to link commits to bug reports (and we already have that in our environment via SVN/CVS/Git commit hooks and Clearquest).
You are correct that the git adapter just links commits to bug reports, so a conversion to GIT would not help in a transition to RTC-SCM.