It's all about the answers!

Ask a question

RTC Branching and Merging Strategy


2
1
Justin Gordon (3646) | asked Nov 01 '12, 8:40 p.m.
  
 
 

RTC Branching and Merging Strategy

By Justin Gordon November 1, 2012

I'm looking for expert advice, suggestions and opinions of this document. My team has differing opinions of how to handle developing for multiple codelines. In the example below, we have Release 1, the first code stream used by customers, Release 2, a newer codestream used by customers, and active new feature development for the next major release on "Trunk".

1 Merge strategy: Each developer concurrently applies forward ports

The following describes the merge process of maintenance releases into newer releases.

  1. Developers are responsible for forward porting their fixes to all branches. This was tracked on a wiki page or RTC tasks.
  2. There was no synchronization of porting/merging the fixes in an orderly manner to observe regressions.
  3. Without any synchronization of order of the code integrations, the RTC tool would have to show it's merge dialog, which is incredibly confusing given that it does not show a clear picture of the actual changes of the changeset being merged. Instead, the merge dialog seems to show all the differences between the different codelines. To make matters further confusing, RTC does not allow two diff windows open at once. The not-so-obvious workaround required is to open two instances of RTC so that one can see the diffs needed.
  4. Without doing the merges in a tool-automated fashion, only manual labor can ensure that all code fixes have been integrated. Conversely, the RTC tool should be tracking which changesets were or were not integrated. Due to the difficulties of the merge tool, many developers manually merged files.
  5. Many times, RTC won't even do a merge and suggests a patch is required. The patch interface is confusing.
  6. New feature (unstable) development occurred simultaneously during the forward port merge process, making it impossible to determine if a defect was occurring due to a merge or new features.
  7. The number of JUnit failures increased to triple digits, rather than being maintained at zero or a very low number.
  8. Simply too many changes were being done simultaneously, making it very difficult to assess the changes.

2 My Proposed Recommendation

2.1 Stable Trunk Codeline

The trunk codeline is the single stable codeline that contains all fixes done to previous releases. See Trunk and Release Codeline Quality Criteria. The key concept is stability. A developer working on feature X should be able to implement it on top of the trunk codeline without being slowed down by random instabilities. The same applies for the developer forward porting changes from maintenance release streams. This makes for a more efficient development team and happier developers.

Fixes are brought from the Stable Release Codelines on a regular schedule, such as every 4 weeks. See Forward Port Strategy for execution details.

Bug fixes are always done on the developer's Code Stream WorkspacesFeatures are always developed on Feature Streams/Workspaces as described below.

2.2 Stable Release Codelines

The release codelines represent the prior major code streams for maintenance releases. These are currently established, shipping codelines. The release codelines should conform to the same quality level as the trunk codeline

Typically, only bug fixes are placed on such codelines. If bug fix is so big that it's really a major feature, then the team should try to avoid implementing such a feature in a release codeline. If this cannot be avoided, then changes should be minimized. I.e., the following should be considered when working in release codelines:

  1. Avoid name changes and other refactorings beyond a small number of lines/files to clarify the code. In other words, renaming a class used by a 100 other classes is a terrible idea no matter how bad the name. Keep ease of merging changes forward in mind.
  2. Avoid any changes unless needed by the bug fix.

The bottom line is to save the urge for doing new, great stuff for the trunk codeline.

2.3 Trunk and Release Codeline Quality Criteria

The below list should be a set of mandatory criteria for the trunk and release codelines, at all points in time. If a developer introduces the issue which causes the below criteria to fail, then the team the team should take immediate appropriate action to address the issue before moving further. To ease the burden on the team and stress on the developer that caused the issue, the code change that caused the issue can be backed out of the stream. JUnit tests that fail due to solely JUnit issues can be flagged  @ignore , and later fixed or deleted as appropriate. Here's the critical list of quality criteria that should apply after each changeset is submitted to the trunk or release stream:

  1. All JUnit tests pass.
  2. UI regression and any other automation tests pass.
  3. The code could be released to a customer on any given day.

2.4 Developer Workspaces

2.4.1 Code Stream Workspaces

For each major code stream, a developer shall have a workspace to be used for bug fixes and to verify the current stream without any pending changes. I.e., before a developer knows if his/her code is causing an issue, the developer should test on the untouched (aka clean) code. Let's call this workspace the code stream workspace. The name of the workspace shall reflect the:

  1. Source code stream
  2. Developer name

2.4.2 Feature Streams/Workspaces

Developers shall make new streams/workspaces for any big features that require:

  1. Significant time to implement.
  2. Shared with other developers.
  3. Higher risk.
  4. A hassle to quickly suspend all pending changes to fix other bugs in the code stream.

Let's call this workspace the feature workspace or feature stream. The name of the workspace/stream shall reflect the:

  1. Source code stream
  2. Developer name(s)
  3. Feature Name

If the feature will be worked on by a single developer, then the developer can create a workspace for the feature. If multiple developers wish to collaborate on a bigger feature, then a stream is needed, as both the developers on the feature will need to accept changes from the feature stream.

2.5 Forward Port Strategy

So long as we can keep a trunk codeline that meets the above quality guidelines, we should conduct forward ports as follows:

  1. A single person, the stream integrator, is responsible for "accepting" changes into the target stream from the source stream, which should typically be the last prior release stream. There should be a well defined path for forward ports to get into the trunk code stream. I.e., if there is a Release 1 and Release 2, and Trunk, it would looks like this, and a change from Release 1 goes only to Release 2, and not to Trunk (under normal circumstances).
    Source Stream Target Stream
    Release 1 Release 2
    Release 2 Trunk
  2. The stream integrator should work with developers that have merge conflicts. Because merges will be done regularly, there will be no extreme urgency to rush multiple merges in simultaneously. Thus the stream integrator can hand off work to the owner of the conflicting changeset, and when the conflicting merge is completed, the stream integrator can continue his work. Because the changesets will be accepted in order, the likelihood of merge conflicts is far lower.
  3. Related to #1, changes should be migrated from one stream to the next higher release, unless there is a special reason not to put an older change in the next release. For example, the Release 1 fixes should be merged into Release 2 and then those changes should get merged into trunk. This means that the Release 2 stream (target) accept changes from Release 1 stream (source), and then the trunk stream (target) accepts changes from the Release 2 stream (source).
  4. Ideally, build validation of the target stream would occur after each work item is imported. Due to practicality, it probably makes more sense to import changes in a batch, and then to "discard" changes in reverse order when the build regresses. For example, suppose that the trunk stream is configured to accept changes from the Release 2 stream, and 5 changes are pending acceptance. After accepting these 5 changes into the merge developer's local workspace, a regression is detected. Changes are backed out until the regression change is identified. The stream integrator may need to consult with the developer that created the problematic changeset, or the stream integrator may consult with the new feature developer that caused the needed change to the code. An example of such a change would be one that involves much code refactoring. This would require manual work in some cases.
  5. These changes should be brought over from the source to the target streams on a regular schedule, such as every 4 weeks. Note, this means that changes should not be deferred until QA tests them. I.e., changes may get integrated with slight bugs. For example, there could be an omission in the developer's fix and a corner case is missed. The developer will then fix the issue on the source stream, and this change will get pulled over to the target on the next regularly scheduled integration.

2.6 Commentary on the Reference Articles

The three articles listed in References are well worth reading. RTC is much more similar to Git than Perforce, as RTC deals with a unit of code change, called a "changeset" which involves diffs combined with the state of the source file. Git has the concept of a "commit" which is a diff relative to it's predecessor. What is similar is that a given "RTC changeset" can exist in multiple streams, just like a "git commit" can exist in multiple branches (branch in git is like a stream in RTC). Perforce has the concept of changelist with its unique ID. Changelists can never exist on multiple code branches/streams.

  1. Stream Strategies with Rational Team Concert
    1. Section titled "Using Side Streams" applies to the above listed concept of Feature Workspaces.
    2. Section titled "Multiple Application Development" applies to developing separate components such as GDS.
    3. We do "Multiple Release Development", done like Figure 3, but using the recommended change direction of maintenance releases to current development.

    Overall, this article provides little suggestive guidance on how to manage the streams. That is why the next two articles are worth reading.

  2. Branching Strategy with Git (can be used for RTC) This article describes the master branch as:

    master should be treated with reverence. The article put it best: 'We consider master to be the main branch where the source code … always reflects a production-ready state.' No changes are made directly to this branch. I [Ben] strongly recommend that at least two sets of eyes see any code that is merged into the master or release branches…

    The point about the develop branch does not exactly apply to RTC, as RTC allows the developer to work in their own workspace. The image shown in the article correctly describes proper procedure of continuously merging changes back in to the "develop" branch. Remember, for RTC, the "develop" and "master" branches can be the same. That is addressed in the next article.

  3. GitHub Flow This article builds on the previous one, with one major distinction. That is to skip having a separate "master" and "develop" branch. While the article is geared toward what is used in a live Software as a Service business (http://www.github.com), I believe that it applies to producing standard released software versions with the addition of having "master" branches for the maintenance releases.

Date: 2012-11-01 Thu

Author: Justin Gordon   

Accepted answer


permanent link
Michael Valenta (3.6k3) | answered Nov 12 '12, 12:32 p.m.
FORUM MODERATOR / JAZZ DEVELOPER
I would say that, in general, your approach looks good. There is a lot there so it is hard to comment on everything. Here are a couple of things that came to mind when I read your post.

Workspaces are lightweight: You mentioned some rules on how users can manage their workspaces and, in general, they makes sense. In practice, I find that users create workspaces as needed. They man have one or several for a paritcular release. Because of the change set model of RTC, it is easy to create a workspace and either accept or discard change sets as needed to get into the desired state.

Forward porting isn't always an option: It is true that RTC has better support for forward porting bug fixes. However, we have found that, in practice, back porting happens more often. For the time being, this is done through patching but improving the support for back porting is high on our priority list (see https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/218181).

Unfortunately, I can't offer any comparison with Clear Case as I have never used it myself.

If you would like more feedback on a particular aspect of what you wrote, feel free to ask.
David Lafreniere selected this answer as the correct answer

Comments
Justin Gordon commented Nov 12 '12, 12:45 p.m. | edited Nov 13 '12, 8:48 p.m.

Hi Michael,

Given that RTC has better support for forward porting and we don't necessarily have a need for back porting, would you say that the approach I present is highly advisable?



Michael Valenta commented Nov 12 '12, 1:21 p.m. | edited Nov 13 '12, 8:49 p.m.
FORUM MODERATOR / JAZZ DEVELOPER

I would agree that, if it is possible, forward porting fixes is going to give you better traceability than back porting with the way things stand feature-wise in RTC today.


David Lafreniere commented Jun 20 '14, 4:36 p.m. | edited Jun 20 '14, 4:37 p.m.
FORUM MODERATOR / JAZZ DEVELOPER

The following information is related to back porting, which was brought up in Michael's response above (which I had to split up into 2 different comments since it was too long)

In RTC 4.0.5 we delivered additional support when trying to accept change sets which have a gap. In these circumstances, you can follow a gap workflow that accepts one change set at a time and, for change sets that contain gaps, creates a new change set that contains the equivalent changes.
This feature is summarized in the RTC 4.0.5 'New & Noteworthy' page:  https://jazz.net/downloads/rational-team-concert/releases/4.0.5?p=news#scm-improve-usability-405-m1
Below are some videos which show this feature:
-Accepting multiple change sets with gaps in the RTC 4.0.5 client for Eclipse IDE: https://www.youtube.com/watch?v=28raag5RdzU
-Accepting a change set with a gap in the RTC 4.0.5 client for Eclipse IDE: https://www.youtube.com/watch?v=TucVu_BgB7E

<continue reading the rest of my comment below>


David Lafreniere commented Jun 20 '14, 4:36 p.m.
FORUM MODERATOR / JAZZ DEVELOPER


In RTC 5.0 we added a "fill the gap" feature where the change sets that fill the gap are shown to the user, allowing them to either accept all the change sets or to continue with the gap workflow that was available in RTC 4.0.5.
This feature is summarized in the RTC 5.0 'New & Noteworthy' page: https://jazz.net/downloads/rational-team-concert/releases/5.0?p=news#eclipse-fill-gaps

Both features are explained in detail in the "Improved Gap Handling for SCM" article: https://jazz.net/library/article/1372

2 other answers



permanent link
w h (191324) | answered Oct 22 '13, 9:25 a.m.
@Justin Gordon  Thanks for sharing this.  We're moving from CC UCM to RTC, and are discussing 'traditional' stream management vs the contemporary styles.  Would it be possible to see your Flow Diagram and / or some thoughts on how this has worked out with your teams?  Thanks!

permanent link
Michael Valenta (3.6k3) | answered Oct 22 '13, 10:00 a.m.
FORUM MODERATOR / JAZZ DEVELOPER
Justin,

That is a very broad topic. One of my team mates wrote a recent blog post that is somewhat related:

https://jazz.net/blog/index.php/2013/08/20/rational-team-concert-source-control-makes-continuous-delivery-easier/

Perhaps that will give you some ideas.

Michael

Your answer


Register or to post your answer.