Since August of 2012, we’ve been working on improving our ability to continuously deliver software. Our work was organized into four different, but overlapping areas — ways of working, continuous deployment, build, and test. The team that I work with is focused on build, the process by which we transform our source code into artifacts that can be deployed and tested.

I’ve personally worked on Rational build technology since before IBM acquired Rational in 2003. In 2009, I led a team that restructured the Jazz builds so that the build of Jazz Foundation was separated from the build of Rational Team Concert (RTC). This work included setting up a build farm with over 70 build engines running on over 50 physical build servers.

Build has a special role in the context of continuous delivery. Traditionally, we run the full integration build daily. This enables continuous integration by integrating the daily contributions of the entire team. Like all production builds, this build creates and tests the bits that we deliver to our customers. But for continuous delivery, the build must be part of a larger, automated process that starts building the customer deliverable and then automatically deploys and tests the product. Being part of the continuous delivery pipeline makes it even more important that all aspects of the build be automated, and that this automation applies to all the situations where the build is used. One of the most important uses for the build is to be a tool that developers can use to validate their deliveries to the integration stream. This is the essence of continuous delivery and DevOps, to give the developer full control over all aspects of delivering value to the customer.

In the context of development of the Collaborative Lifecycle Management (CLM) products, we have many challenges. Our teams are globally distributed with active development in 17 different sites in time zones that span the globe. Our projects are large, both in the number of developers and the size of the products. Our largest project is the development of Rational Team Concert (RTC) which has 98 contributors. RTC is a big product, consisting of clients and servers for many platforms. Each build of RTC produces 47 GB of data. The cumulative size of a single build of the complete set of CLM products (Rational Team Concert, Rational Requirements Composer (RRC), and Rational Quality Manager (RQM)) is 103 GB! Although our projects are large, we continuously integrate all of the CLM products. Every day we build all of the CLM products in a single build that picks up the contributions of four interdependent product teams — Jazz Foundation, Rational Team Concert, Rational Requirements Composer, and Rational Quality Manager. These builds automatically accept and integrate the latest deliveries from each of the product teams. This complexity is multiplied by the need to do development and support on multiple release streams — 2.x, 3.x, and 4.x.

Over the years we’ve developed and adapted our build technology to meet these challenges. We’ve measured our builds to highlight problem areas so that we can deal with them effectively. Our measurements have focused on build speed and reliability. In the past we’ve focused on the speed and reliability of the production builds.  We’ve made excellent progress by increasing automation, parallelism, and re-use. A year ago, a complete build of all of CLM took over 11 1/2 hours. We’ve cut that time in half. But we’ve come to realize that improvements in the speed of the production builds do not necessarily translate into benefits to the developer. We make great progress in speeding up our production builds, but we’d done little to empower our developers. This has led us to look more closely at the relationship between build and the development process.

Here’s a high level description of the development process:

In preparation for making changes, a developer must set up an environment that matches the contents of the integration stream — both in terms of source code and for any components on which the product depends. For example, a developer making a change to RTC must make his personal workspace match the source code used by the RTC integration build. He must also set up his development environment to contain the same versions of components from Jazz Foundation that the RTC integration build depends on.

Once his environment is set up, our developer makes his changes — he edits the source code, builds the artifacts he needs for testing, and runs tests. He iterates on this process until he thinks his changes are complete. We need to make the change/build/test cycle time for the developer to be as short as possible. Earlier, I noted that we have three tricks for speeding up full production builds — automation, parallelism, and re-use. In this context, though, there’s another optimization — to make it easy for developers to limit the build the places which have changed recently and any related components. For changes that affect consumers, a developer may need to not only build his own components, but also perform test builds of any code that depends on his components.

At this point, our developer needs to reliably deliver his changes to the integration stream. This means that the developer must be able to validate that a delivery will not break the next build of the integration stream. RTC supports this process with personal builds, which enable developers to perform an integration build from their workspace.

Everything described above is captured in the use case of fixing a bug. Here are the specifics of the use case:

A customer reports a bug against a specific version of a product. Based on the bug report, the developer then configures his environment to match the state of the source code and any required components that were used to create the release. In the context of this environment, he debugs and fixes the problem. He then builds, deploys, and tests the complete product which contains his fix. If the tests pass, he delivers the new code to production. From here, the automated builds create and deliver the fix to the customer. We can measure the quality of our build technology by the time it takes to perform this use case.

Continuous Delivery and DevOps are changing our notions about the role of build. The build has expanded from a tool to produce the customer deliverable to also be a tool to empower developers. This has caused us to think about builds in a new way. We’ve engaged with our development community to better understand their requirements. In future blogs I’ll talk about that engagement and the changes it’s bringing.