Managing vendor/third-party code in Rational Team Concert source control
Summary
When developing, we often find we need to make use of code from another vendor or third-party. This article describes how to use Rational Team Concert’s (RTC) Source Control when working with Vendor/Third-party code.
Table of Contents
- Introduction
- Stream Setup
- Import Vendor/Third-party code into RTC Source Control
- Making application specific changes to Vendor/Third-party code
- Integrating a newer release of Vendor/Third-party code
- Answering the concerns about Vendor code
- Know what version of the vendor code was used in a particular release of our application
- Make sure all our team mates are using the right version of the the third-party code
- Be able to fix/extend the third-party code and manage those changes
- Integrate newer releases of third-party code easily
- Integrate newer releases even when we have fixed/extended the earlier version of the third-party code
Introduction
When developing, we don’t want to spend time re-inventing the wheel. We often find we can make use of code from other vendors. The vendor could be a business partner, another team in the organization, open source, etc. Either way, making use of third-party code allows us to efficiently and effectively develop our application. Of course, the third-party code may not be perfect. We will probably want to fix blocking bugs (at least until the vendor fixes it in the next release) and we might want to make some “enhancements” to the code to provide some additional functionality.
Initially, one might think, “No problem. I will get the third-party source code, and just change it to what I need.” But in real life, that just won’t scale with the lifecycle of the application being developed. We need to:
- Know what version of the vendor code was used in a particular release of our application.
- Make sure all our team mates are using the right version of the third-party code.
- Be able to fix/extend the third-party code and manage those changes.
- Integrate newer releases of third-party code easily.
- Integrate newer releases even when we have fixed/extended the earlier version of the third-party code.
Sounds like a lot of complications and an argument for re-invention. But in actuality with RTC Source Control it’s not that complicated at all.
So, how do I set things up to easily do this? Basically, we need to get the Third-party code into RTC and then setup a workflow that describes an optimal way to work with it.
Stream Setup
The recommended way of setting things up is to create a stream with a component for the third-party code to be used. If you are getting multiple subsystems from the same vendor, each subsystem that has its own release should go in a separate component. The stream will track the original versions of the third-party code. When a new release of the code arrives, it will be delivered to this stream. For the remainder of this discussion, this stream is the vendor stream.
The application that we are developing will also have its own stream with components containing our work. This stream will also contain the third-party code component. This is the stream that we should deliver additional modifications of the third-party code to. So essentially, it will contain the third-party release plus some additional change sets that give us our desired behavior. This stream for the remainder of the discussion will be known as the development stream (to distinguish it from the vendor stream).
As an example, suppose the JUnit team decides that they want to introduce a new feature where the JUnit results are shown in an intricate set of reports. To do this they are going to make use of some sweet reporting capabilities from a third-party. Their stream setup would then look like this.
There is a vendor stream for the Sweet Reporter vendor/third-party code. It has no special modifications to it, just the pure release. Markus has a workspace that he uses to deliver new releases of the Sweet Reporter to the Sweet Reporter (vendor) stream.
There is also a development stream for the JUnit project. The Sweet Reporter component is also present in that stream. The JUnit stream’s Sweet Reporter’s configuration may have additional change sets containing development specific changes. Markus also has a workspace that he uses to integrate new releases of Sweet Reporter (from Sweet Reporter stream) with the development specific changes his team had made to Sweet Reporter (from JUnit stream). He delivers the integrated result to the JUnit stream.
Import Vendor/Third-party code into RTC Source Control
You want to keep each Vendor’s code segregated so that it can be easily updated when newer versions are available. So it should be placed in its own component within its own stream. How to import the third-party code into RTC depends upon where the source code is located.
If the third-party code is in another SCM system (i.e. Subversion, CVS, ClearCase), you can use RTC Connectors and tools to import the code into RTC and then later as new releases become available use the same tools to synchronize the Stream with the other SCM system.
If you have just plain source, you would create a stream and corresponding workspace with a single component to hold the third-party code. You would then share the projects into the repository workspace. When done, deliver a new baseline to the vendor stream. You can label the baseline and supply a description with it describing what release it is that you are delivering. Now the Vendor/third-party code is in RTC.
In our example Markus on the JUnit team has received the sweet reporter functionality in the form of a zip file.
The source code is shared into RTC Source Control. Markus created a new baseline for the Beta 1.0 release received and is now about to deliver it from his (vendor) workspace to the Sweet Reporter (vendor) stream
Once all the Vendor/third-party code is in RTC, the next thing to do is make it available to all your team mates. There are a number of ways to do this. You could with your workspace still loaded, change the flow target to point to your Team’s development stream and just deliver the component addition.
The Sweet Reporter component is an outgoing component addition for the JUnit development stream. The JUnit is incoming because this is the vendor workspace which doesn’t have the JUnit work in it. Markus would only deliver the Sweet Reporter component.
If you are a bit more cautious you might want to see just how it all fits together with the rest of your code before springing it on your team mates. You might want to even include it in a build first. If this is the case, an alternative way would be to start with a repository workspace containing just your application’s code loaded. Change the flow target of that workspace to point to the vendor stream. You will see an incoming component addition for the component containing the third-party code, accept the addition.
Markus’s development workspace has the Sweet Reporter component as an incoming component addition (from the Sweet Reporter vendor stream). The JUnit component is shown as an outgoing component because it is not in the vendor stream. Markus will only accept the incoming component addition (he doesn’t want to deliver JUnit because it is not vendor code).
Once you have accepted the component addition, change your flow target back to the development stream. Now you can run a personal build, do any immediate integration changes needed, etc. When you are ready to share the changes with your team, deliver them to your stream.
Markus is going to deliver the SweetReporter component to the JUnit development stream. Markus might also decide to change some of the process rules for the Sweet Reporter component. Such as not enforcing copyright rules (Its not our code so it should not have our copyright notice in it), etc.
Making application specific changes to Vendor/Third-party code
You will implement your features that use the third-party code in separate projects that are in a development component. However, at some point in time you might discover that you need to make changes to the third-party code itself.
You have the third-party code in your workspace and can make changes to it the same way as you do your application. Ideally you will have work items associated with the change sets describing if they are temporary fixes, or enhancements made to the code. While you know exactly what are the changes right now, later on you or someone else will have to think about porting them to a newer release of the third-party code and having the details captured will be helpful. Once your changes are done, you should deliver them to your Team’s development stream. You should NOT deliver them to the vendor stream. The vendor stream is intended to contain the pure version received from the vendor.
You are probably wondering why the component configuration is diverging between the vendor stream and the development stream. The main reason is to easily isolate the application specific changes made. The only time you work with the vendor stream is when you accept newer releases of the third-party code.
Integrating a newer release of Vendor/Third-party code
Integrating a newer release of Vendor/Third-party code is a two step process. First you need to import the newer release of the third-party code. Then you need to integrate it into your development stream.
Importing the new release of Third-party code
If the third-party code came from another SCM system (i.e. Subversion, CVS, ClearCase), you can use RTC Connectors and tools to synchronize the vendor stream with the other SCM system. This will result in a baseline/change sets that describe the changes made to go to the newer release of the third-party code. Make sure you have a new baseline delivered to the vendor stream describing the vendor release it corresponds to.
If you have just plain source code it’s a little more complicated. You want to know the delta between releases so that we can compare and merge in the application specific changes made.
You need a RTC repository workspace with the vendor stream as the flow target. The workspace should be caught up with the vendor stream (i.e. no incoming/outgoing changes). The repository workspace should not be loaded. Put the new release of the third-party code in an empty sandbox (local workspace). That is, no repository workspace is loaded. Now share the projects with the vendor component in the workspace. This is sometimes called “re-sharing”.
Markus has unzipped the newest Sweet Reporter release in an eclipse workspace. He is now going through the Share wizard and will share the projects to the Sweet Reporter component in his vendor workspace.
This will result in a set of unchecked-in changes being created. These changes reflect the changes that have been made to the earlier release to produce the current release. They should be checked into the repository. This process will not identify the projects that have been deleted. You will need to load those from your vendor workspace, select them in the project explorer and from the context menu, select Team > Delete from Repository. This will create the necessary changes to delete the projects.
Because change sets are created with the differences between releases, you can look at the change set and see what is the overall delta of the new release. You can explore see the actual changes made. Furthermore, over time, you can see the history of the files, use annotate to determine how and when they have changed, etc.
Markus has a change set that describe the changes that were made to release Beta 1.0 of Sweet Reporter (to form release Beta 2.0). He can see some new files were added in the public API. He can open the compare editor to see the api changes.
Once all the changes are checked in, create and deliver a new baseline for the new release to the vendor stream. Markus will be delivering the Beta 2 baseline to the Sweet Reporter vendor stream.
Integrating into the development stream
If you did not make any application specific changes, this step is simple. If you had to make application specific changes, you will need to get the necessary ones re-applied to the new release. This entails forward porting the changes.
The examples below range from the simplest case to more complicated cases. They are mainly intended to show you how you can forward-port changes using a variety of RTC Source Control features.
Integrating when there are NO application specific changes
Within a loaded development workspace, change your flow target to point to the vendor stream. Accept the incoming baseline from the vendor stream. Change your flow target back to your development stream. Once you are sure everything works fine with the new release, share the new release with your team mates by delivering the baseline to your development stream.
Markus accepts the new release (incoming baseline) from the vendor stream. Notice when Markus changed his flow target, rather than changing the flow target for the whole workspace, he just changed the flow target of the Sweet Reporter component.
Markus changed his flow target back to JUnit and is now delivering the new release (outgoing baseline) to the development stream.
Integrating when there are application specific changes
You will need to look at your application specific changes and decide which ones are no longer required. There could be changes that are no longer required because the issue has been fixed in the newest release.
How you integrate will depend upon the volume of changes you have made, whether the changes are to the same resources that have changed in the new release, whether the changes you still need have been built on top of changes that are no longer needed, etc. These are general issues that apply with any forward porting that has to happen. RTC Source Control provides features for helping you work through it. We will go through some of the situations that you might encounter and how you could work through them. It is strongly recommended that you read the articles on Multi-stream development (porting in particular) and Conflicts first.
All our changes must be ported
Markus has found that the new release of SweetReporter has some improvements, but does not address any of the issues that the JUnit team had with it. All the changes that JUnit team made to it are still required. Luckily the changes in the new SweetReporter release do not affect any of the changes the JUnit team made. He doesn’t see any potential conflict markers in his pending changes view
Markus has his development workspace loaded. In his pending changes view he has configured the flow target for the Sweet Reporter component to be the vendor stream. He has an incoming baseline for the new release. The outgoing changes are the fixes the JUnit team made for the Sweet Reporter component.
Markus accepts the new release into his workspace and tests it out. Markus changes his flow target for the SweetReporter component to point to his development stream.
He delivers the new release to the development stream. The stream contains the new release plus the fix for work item 61.
All our changes must be ported – some overlap on resources affected
Markus has found that the new release of SweetReporter has some improvements, but does not address any of the issues that the JUnit team had with it. All the changes that JUnit team made to it are still required. Some of the changes in the release affect the same resources that the JUnit team had to modify for the fixes.
Markus has his development workspace loaded. In his pending changes view he has configured the flow target for the Sweet Reporter component to be the Sweet Reporter vendor stream. He has an incoming baseline for the new release. The outgoing changes were the enhancements and fixes the JUnit team made for the Sweet Reporter component. He can see that he has potential conflicts on the Data and ReportDefinition files.
Markus realizes he is going to have to merge the team’s fixes for the SweetReporter component into the new release. Markus has read the conflicts tutorial and has decided that since he knows the JUnit team’s changes the best, he is going to do an inverse merge. That is he is going to merge (apply) in the changes that the JUnit team made to the new release.
Markus replaces his Sweet Reporter component in his development workspace with the latest from the Sweet Reporter stream. He then changes the flow target of the Sweet Reporter component back to his development stream.
After Markus accepts the fixes from his development stream, he will need to resolve the conflicts. The incoming conflicting change that has to be merged in is the change made by his team. He understands them (or can get details on them from the work items) and can rework the changes into the third-party code.
Once the conflicts are resolved, Markus can test out the JUnit application, run a personal build etc. Once he is satisfied with the release, he can deliver to his development stream.
Some of our changes need to be ported – some overlap on the resources affected.
Markus has found that the new release of SweetReporter has major improvements. In fact some areas have been completely rewritten. Some of JUnit’s issues with it have been resolved while others have not. The JUnit team has made a number of fixes to the Sweet Reporter and with this new release only some of the changes need to be ported.
Markus has read the conflicts tutorial and the multi-stream development guide especially the sections on porting. He plans on doing an inverse merge and choosing just the change sets with fixes that he needs. After looking at the changes he has decided that the fix made for work item 64 is nolonger needed. That problem was fixed in the new release. The changes made for work items 61 and 62 however are not included (maybe in the next release).
Markus replaces his SweetReporter component in his development workspace with the latest from the vendor stream and has set the flow target of the SweetReporter component back to be his development stream.
So far the setup is the same as in previous example. This example differs in that not all the change sets will be ported forward on the new release.
Markus decides he needs the fixes for work items 61 and 62 but not 64. Work item 61 is an independent fix. Work item 62 however affects resources that were modified to fix work item 64. Essentially what he wants to do is discard the change set for work item 64. Markus has read the FAQ about how to discard from a stream and realizes he has a few choices.
He could reverse the change set for work item 64 and then merge all the change sets (61, 64, 62 and “64 : Reversal”) into the new release. This makes the most sense if:
- Other team mates are currently working on fixing other issues with the current release that touch the same resources and will likely need to integrate the fix into the new release.
- There are a lot of change sets still required that build on resources touched by change set for work item 64.
To follow this approach, prior to replacing the Sweet Reporter component with the new release, Markus would create a reversal for change set 64, merge in the patch, check in the pending changes into a new change set. This change set is the reversal change set. He will complete the change set but since he is not ready to put it in the development stream, he would suspend the change set. He would then replace his Sweet Reporter component with the new release baseline.
Markus then accepts all the incoming change sets (fixes) his team made to the release. He also resumes the reversal change set. He would then resolve the conflicts by merging his team’s work into the new release. Once he is satisfied with the results he would deliver the new release, reversal change set and the merges to the development stream.
Markus could also decide to just accept the change set for work item 61. When he accepts the change set for work item 62 he will be asked about creating a patch because not all change sets leading up to it are present (we did not accept the change set for work item 64). He will merge the patch into his workspace and check-in the changes into a new change set. This makes the most sense if:
- Most of his team’s previous fixes are no longer required. He does not want to have to deal with the merging with the non-changes (although they will most likely be auto-resolvable since there is no change) as he gets new releases of SweetReporter.
If Markus goes this route, he will need to replace in his stream the new baseline that contains the integrated fixes. He will also want to warn his team mates that after he delivers they may see an out going changes for work item 62 and 64 and what they should do is simply replace the SweetReporter component with the latest from the development stream.
Answering the concerns about Vendor code
Back at the start we identified some concerns about making use of vendor code. In summary we will illustrate how the concerns are covered with the use of RTC Source Control features.
Know what version of the vendor code was used in a particular release of our application
The vendor code is isolated within a component in the development stream. When the release is declared and you create a snapshot of the development stream, the snapshot will contain a baseline for the vendor component. You can compare that baseline against the vendor baseline and see exactly what additional changes were made as well.
Make sure all our team mates are using the right version of the third-party code
The vendor code is in the development stream. All our team mates will see incoming changes from that stream when new releases are made available. They work with the vendor component in the same way as they work with the development components (except most will likely never have to deliver changes to it).
Be able to fix/extend the third-party code and manage those changes
Changes made to the third-party code are checked into change sets. The change sets can be tracked via work items, reviewed, etc. The original vendor code releases are tracked by baselines. We can easily compare a configuration to a baseline to determine the changes made.
Integrate newer releases of third-party code easily
RTC provides functionality that synchronizes with other SCM systems. It also provides a way to re-share resources with an existing component. Once the new third-party code release is in a workspace, you can test; run personal builds; etc. prior to sharing the release with your team. Integrating a new release can be as easy as accept, change flow target, deliver.
Integrate newer releases even when we have fixed/extended the earlier version of the third-party code.
The new releases of the third-party code added to RTC as a set of change sets that describe the delta between the releases. The result is we have a history of changes made to the third-party code. The application changes made are also recorded in change sets. This allows the merging, conflict resolution and patch creation to be done against third-party code, the same way as you would with application code. The third-party code is not treated as a second class citizen in Rational Team Concert.
About the author
Heather Fraser-Dubé works for IBM and is a member of the RTC Source Control development team
© Copyright 2008-2009 IBM Corporation
Yes