Chris Daly’s entry “Some Notes on Rhythm” provides a good high-level overview of how the Jazz Project establishes a “rhythm” to make consistent and predictable progress over the course of project milestones and release cycles. In this entry I’d like to drill-down to show how we use our tools to keep a rhythm for day-to-day development. Specifically, I’ll talk about the process that the Jazz Repository team uses to manage both our team streams and our deliveries to the project’s integration streams.

But first, a word about the goals of the Jazz Platform’s Repository component. The Repository component sits below all the other components in the Jazz stack (see diagram below), and provides fundamental capabilities (like the ability to save and fetch items from the database) that upstream components build upon. The biggest problem we have hit is trying to balance the stabilization of code we have previously delivered while still making forward progress on new features. The other teams are trying to do the same thing, and usually want to see a preview of the new features so that they can plan and adjust for things coming.

We’ve addressed this problem using Rational Team Concert’s SCM capabilities, which provide support for multiple streams and component baselines. At the Jazz project level the main stream is the Weekly Integration stream, which tracks the most stable version of every component for that week, and the secondary stream is Continuous Integration, which tracks each team’s latest (but pretty stable) code. Each team’s baselines, which capture the the stable state of a component (similar to a CVS tag), flow from the Continuous Integration stream to the Weekly stream each Monday and code is stabilized throughout the week until each team is happy with the build. During the week bug fixes flow into the Weekly Integration stream, and new features flow into the Continuous Integration stream for early testing and adoption from other teams. The balance of the two different goals is maintained by having the Weekly stream be geared towards stabilization and the Continuous stream geared towards new features and ongoing development and non-critical bug fixes.

When I’ve completed work on a new feature for the Repository component, I deliver my change-sets to the Repository development stream and eventually those change-sets flow into the Continuous stream for other teams to see and use. Each team decides how often they want to deliver the code from their development stream to the Continuous stream, and their choice could depend on how stable they think their development stream is, or how much people want to look at their newest features. For the Repository component, we have tried to push our code to Continuous almost daily so teams can leverage new features quickly, and so we can get good test coverage on low-level changes even before the Weekly stream is updated.

If a bug is found during the testing of the weekly builds, then the situation gets a little bit trickier. I need to get the version of our component that was used in the weekly build so that I can create a fix that doesn’t depend on any new changes, and eventually I need to get my change into the Weekly stream so that a new build can be started. On the Repository team we have solved this problem by adding a new stream into the mix, the Repository Patch stream, which tracks our latest contribution to the Weekly stream and provides a place to gather bug fixes for testing before a new baseline is delivered to the Weekly stream. So, to fix the bug I simply grab the latest code in the Repository Patch stream, fix the bug and deliver my fix back to the Repository Patch stream. The fixes that get put in the patch stream will flow back into the Weekly stream when they have been tested and all the pending fixes are ready.

The last part of the problem is: how do all of the changes magically flow from stream to stream? If I deliver a bugfix to the Patch stream, how do I know that I didn’t break something before it gets put into the Weekly stream, or if I deliver something to the Repository stream, how does it eventually make it’s way to the Continuous stream? Our solution to that is to have a release engineer role which we rotate among team members. Whoever is currently the release engineer manages moving our code from stream to stream on schedule and makes sure that our new deliveries don’t introduce any problems. Since everybody takes turns wearing the release engineer hat, you only have to focus on the process every couple of months, and we benefit from the fact that everybody becomes familiar with the process and we don’t become overly-dependent on a single rel-eng expert. In addition to learning the process, the release engineer becomes more familiar with Jazz SCM beyond simple ‘accept’ and ‘deliver’, such as managing baselines and working across multiple streams.

To summarize, and for visual learners, here is a diagram that shows the details or our daily flow of changes between each of the streams from the Repository perspective; the dotted-lines represent change-set flows while the solid lines represent baseline flows.

—
Matt Lavin
Jazz Repository Team