Continuous Integration Best Practices with Rational Team Concert
Summary
Continuous integration is a software development best practice that distributed teams use more and more as a way to mitigate integration problems and to facilitate development of cohesive software more rapidly. The objective of this technical article is to introduce briefly continuous integration key practices (referencing Martin Fowler “Continuous Integration” white paper and explain using a practical scenario how Rational Team Concert (RTC) can help software development teams easily build a Continuous Integration system.
Continuous Integration Key Practices
The effort required to integrate a system increases exponentially with time. Continuous integration is a software development practice that promotes frequent team integrations and automatic builds. By integrating the system more frequently, integration issues are identified earlier, when they are easier to fix, and the overall integration effort is reduced. The result is a higher-quality product and more predictable delivery schedules.
Continuous integration provides the following benefits:
- Improved feedback: Continuous integration shows constant and demonstrable progress.
- Improved bug detection: Continuous integration enables you to detect and remove errors early, often minutes after they’ve been injected into the product.
- Improved collaboration: Continuous integration enables team members to work together safely. They know that they can make a change to their code, integrate the system, and determine very quickly whether or not their change conflicts with others.
- Improved system integration: By integrating continuously throughout your product, you know that you can actually build the system, thereby mitigating integration surprises at the end of the lifecycle.
- Reduced number of parallel changes that need to be merged and tested.
- Reduced number of errors found during system testing: All conflicts are resolved before making new change sets available and by the person who is in the best position to resolve them.
- Reduced technical risk: You always have an up-to-date system to test against.
- Reduced management risk: By continuously integrating your system, you know exactly how much functionality that you have built to date, thereby improving your ability to predict when and if you are actually going to be able to deliver the necessary functionality.
The referenced white paper lists the following key practices that make up effective Continuous Integration:
- Maintain a Single Source Repository.
- Automate the Build.
- Make Your Build Self-Testing.
- Everyone Commits To the Mainline Every Day.
- Every Commit Should Build the Mainline on an Integration Machine.
- Keep the Build Fast.
- Test in a Clone of the Production Environment.
- Make it Easy for Anyone to Get the Latest Executable.
- Everyone can see what’s happening.
- Automate Deployment.
While there are many ways to implement above Continuous Integration key practices in your development environment, we believe that Rational Team Concert offers an excellent platform on which to base these practices. Rational Team Concert (RTC) is a team collaboration tool that is built on a scalable, extensible Jazz platform that integrates tasks across the software lifecycle. Rational Team Concert integrates work item tracking, source control management (SCM), continuous builds, agile planning, and project transparency (dashboard and reporting) components into Jazz Team Server:
Figure 1.- Rational Team Concert Components
Using Rational Team Concert integrated components you are able to implement easily and effectively a Continuous Integration system that will help you to achieve the benefits outlined above. If you want to know more technical details about the product, please go here.
Let’s review in following sections how RTC helps us to cover those Continuous Integration key practices.
Note: We will use JUnit example project to demonstrate Continuous Integration capabilities of RTC.
Continuous Integration Key Practices with Rational Team Concert
1. Maintain a Single Source Repository
The base of a Continuous Integration system is to implement a good source control management system to keep track and control all of the files needed to build a product. In this source control repository you must include everything you need for the build.
RTC implements a Source Control component which manages the source code, documents, and other artifacts that a team creates. It provides change-flow management to facilitate sharing of controlled artifacts, retains a history of changes made to these artifacts, and enables simultaneous development of multiple versions of shared artifacts, so that teams can work on several development lines at the same time.
Source Control component organizes versionable items (files and folders) into components and streams, and provides workspaces where you can view and modify file and folder contents. Together, these repository objects represent the configuration of the system being developed and allow any configuration to be retrieved, shared, or built. They organize a team’s files, track and share changes, and keep the entire team working in concert to achieve common goals.
The following figure shows how change sets created in a local workspace and then checked in to a repository workspace can be shared between different team meambers by delivering them to one or more streams that all team members use:
Figure 2.- Rational Team Concert change sets flow
Each user may have his own private repository workspace which stores the changes he has made, regardless of when he decides to make them available to the team. When the user loads his repository workspace, the files and folders in it are transferred to his local private Eclipse workspace on his computer. To push a change from the Eclipse workspace to the repository workspace the user checks-in the change.
A stream is used to store the team’s work. Each team may have one or more streams to share different releases. When a user wants to make his changes available to the team, he delivers them from a repository workspace to a stream. Changes can be committed to the repository workspace separately from deliverying, offering a kind of server backup for the private files. When the user wishes to incorporate other team members’ changes, he accepts them from the stream.
All changes the user makes in his repository workspace are tracked within change-sets. Each change-set is composed of a collection of explicit, primitive changes to one or more files or folders. A change-set can also carry a comment and the reason the changes were made, typically by referencing the relevant work item.
Rational Team Concert provides additional features to manipulate easily change sets in the workspace, offering to the user simple operations to suspend, reverse and/or discard change sets.
In our example, we illustrate how a development team is using a main development stream called JUnit with a single component also called JUnit.
Figure 3.- Source Control artifacts
This single component includes source code, unit tests, build elements, etc… but the Source Control component is very flexible allowing you to define and create several development streams (development, maintenance, integration, etc…) and several components (source code, tests, libraries, etc…). The choice is a matter of how you decide to organize your development team. Try to reduce the number of streams to a minimum and always have a main development stream to integrate easily team member changes.
Figure 4.- Example of component content
It is very important for team success to minimize the time spent to set up the development environment for a new developer who arrives to the team, or to recreate the environment of a specific version giving problems in production. The ability to achieve this is one of the most important aspects in development. Any team member should be able to get quickly the last version of files and executables and be able to run them.
With a correct configuration of components (source code, libraries, build scripts, etc.. adjusting all dependencies.) and associating them to the development stream, it is very easy and quick to arrive as a new developer to a project and recreate the necessary environment. It is simply a matter of connecting to a Project Area within the RTC server, and creating a Repository Workspace flowing to the development stream. If that stream has been associated to the correct files, then automatically all needed code will be loaded to developer’s local workspace.
Figure 5.- Example of Repository Workspace creation
Figure 6.- Example of a Repository Workspace flowing to stream
More info about the Source Control component can be found here.
2. Automate the Build
To get an efficient Continuous Integration system you need to implement an automatic build process. The details of this step depend upon the implementation language and development environment and may involve compiling and linking and/or other processes that produce the build artifacts, such as the program executable.
It is important that you include all steps in the automated build, including for example: compiling and linking the source artifacts to create an executable, loading binary objects on a test bench, running a script to load/update database schemas, and packaging and deploying web applications.
Try to reduce time needed for automatic builds, analyzing what has been changed from the previous build and what has to be rebuilt.
RTC implements a Team Build component which provides support for the automation, monitoring, and awareness of a team’s regular builds. This component provides a model for representing the team’s build definitions, build engines, and build results. The model supports teams with different build technologies.
It includes the Build Engine and an Ant build toolkit that can publish build information to the RTC repository. The Ant build toolkit can perform various operations on a build, such as indicating build progress, publishing built artifacts (i.e. downloads) and build logs, and publishing compilation and test results. While the build toolkit is best suited for Ant builds, you can use any scripting technology. For example, a team can use Perl, DOS batch files, Maven or Make, to create build scripts that interact with the Team Build component.
A team typically has a build system that runs builds from a dedicated build machine. The RTC Server is situated between the developers and the build system. The following generic build-related items are stored in the RTC repository and support the build process:
- Build definition: Defines a build, such as a weekly project-wide integration build.
- Build engine: Represents a build system running on a build server.
- Build request: Represents a request to run a build.
- Build result: Represents the output from a build.
All build-related items belong to a project area. Build-related operations are governed by the project’s process.
Figure 7.- Team Build artifacts
You can use the RTC clients (Web, Eclipse and VisualStudio) to perform the following tasks:
- Submit a request to run a build definition.
- Check the build status.
- View completed build outputs, such as logs, downloads, external links and compilation and test results.
Figure 8.- Example of build execution history
Figure 9.- Example of a build result
More info about the Team Build component can be found here.
3. Make Your Build Self-Testing
A good practice that will help you to detect errors quickly is to include some automated tests in your build process. If possible, execute a sub-set of the system tests as part of the build to ensure that the build is suitable prior to committing resources to the full scope of system testing. While the level of testing may vary, focus on gaining confidence that the increment is of sufficient quality to establish a baseline for system testing. If some testing fails, the build should also fail.
RTC is not an IDE, so it doesn’t provide features to define and implement unit tests. But in its Team Build component it includes support for execution and result analysis of different unit test frameworks like: CPPUnit, JUnit, MSTest and NUnit. So development teams will be able to execute and check the results of all the unit tests of the project.
Figure 10.- Example of JUnit tests execution as part of a build result
More info about how to publish unit testing results to the Team Build component can be found here.
4. Everyone Commits To the Mainline Every Day
This is more a methodology aspect than a technical issue to be resolved or supported by the tool. Communication is key for continuous integration, and developers communicate with others delivering changes they have made to their files. A good practice is to force developers to commit their changes to the main development stream at least once every day; before delivering the files, each developer must verify that his/her working copy is consistent with the main development stream, resolve conflicts if they exist, build a local test, and if this passes, commit changes to the main development stream.
Of course, the RTC Source Control component lets the developer to commit or deliver code to streams as many times as they require as needed (see Figure 2). So a development team adopting Continuous Integration practices should encourage its developers to commit at least once per day.
The RTC Build component includes a feature that could be very interesting for development teams. It is what is called a personal build.
A personal build lets a developer execute a build against his/her personal workspace instead of the stream, so he/she can know if his/her code will break or not the build before committing any code change to the stream. The rest of the team won’t be notified of these kinds of build executions.
Figure 11.- Example of a personal build execution
More info about the Source Control component can be found here.
5. Every Commit Should Build the Mainline on an Integration Machine
Although developers must run local builds and tests in their local machines before delivering to the main development stream, there may be differences between each developer’s machine, or code integration errors. This is why it is very important to ensure that an integration build is run on an integration machine each time each a developer commits some changes. The developer that delivers the changes should be responsible for monitoring this integration build and fix it if there is any problem. Working this way ensures that errors can be found and solved easily and quickly. Developers are expected to commit to fixing problems with the build quickly, resulting in a successful build every day.
As commented by Martin Fowler in his white paper, there are different approaches to cover this aspect.
The first commented approach is to let the developer manually request a build execution after he has committed some new code to the stream.
The RTC Team build component enables developers (if they have permission) manually to request a build execution. It doesn’t matter the type of client he is using: Web, Eclipse or VisualStudio.
Figure 12.- Requesting a build execution manually
A second approach is to ensure that an automatic build will be executed after a code commit to the stream.
As we have commented already, RTC includes different components such as the Source Control and Team Build components. By default, RTC already implements automatic linkage between both components. When the developer creates a new build definition, the Jazz Source Control page of the build definition lets him set if the build goes ahead based on whether new changes are available in the stream. Combined with the build feature of automatic scheduling of periodic builds, RTC can take care of running a build whenever there are new changes. Once the build is finished, RTC will create an automatic "included in build" link for the work items associated with the accepted change sets.
You can also implement custom integrations between Source Control and Team Build components easily based on the concept of Follow-up actions. Follow-up actions are logic statements executed after a specific event has occurred in the server. So the main idea here is to attach a Follow-up action which requests a build execution to stream commit event.
RTC includes a large number of already implemented Follow-up actions. You could easily for example create a “Request Build Execution” Follow-up action to ensure that an automatic build is executed after some code is committed to the stream.
The RTC server is built on different modules that technically could be Eclipse plug-ins or OSGi bundles and it includes a SDK to help us to implement such extensions to RTC.
More info about the RTC SDK can be found here. There is also a newsgroup dedicated to RTC extensibility that can be found here.
Developing RTC extensions is out of the scope of this article, but all required details can be obtained from links provided above.
Figure 13.- Using the custom Follow-up action
A third approach is to use a scheduler so that automatic builds can be executed daily, weekly basis, etc…
The Team Build component implements a scheduler to make it very easy to setup such automatic build execution, for example, a nightly integration build (as used in projects such as Eclipse.org). The scheduler is very flexible.
Figure 14.- Example of build schedule
More info about the Team Build component can be found here.
6. Keep the Build Fast
It is important to reduce to a minimum the time spent running the integration build each time a developer commits changes to the main development stream. The problem is that this is not always possible, and complete integration builds can take a lot of time and resources. If this is your problem, a good best practice suggested by Martin Fowler is to split your integration build into different stages: create a commit build that compiles and verifies the absence of critical errors when each developer commits changes to the main development stream, and create secondary build(s) to run slower and less important tests. You need to ensure that commit builds are always successful, but if any secondary build fails it does not have to be so critical as to stop everything. This allows your developers to continue working while these secondary errors are fixed.
RTC can’t help too much in this aspect. It is a matter of how well the build script is designed and written. What the Team Build component offers that can be very useful to detect time consuming areas of the build is information of the duration of each activity. So developers in charge of the build script can detect possible areas to improve in the builds.
Figure 15.- Example of build activities executed with associated duration
For speeding up continuous build feedback, each sub-team may have their own stream and run continuous builds until they are free of critical errors, and then deliver to a higher level integration stream to run a complete team build.
It is also possible to split builds into several smaller builds, extracting steps or activities for the basic build that can be postponed to later as part of a more complex and complete build. As a result, developers in their daily work can execute just a subset of the final build.
RTC Team Build lets you define and execute many builds, and those builds can be related so one build can invoke others, etc…
More info about the Team Build component can be found here.
7. Test in a Clone of the Production Environment
Although many times it is difficult to achieve, you should have a test environment as similar as possible to your final production environment.
Once again, this is more a methodology aspect than a specific feature in RTC.
However, it is worth noting that the RTC Build Team component can be installed multiple times in multiple different environments (all installations working against the same RTC server). Therefore it is very easy to setup several build environments (development, integration and production) executing the builds with different scheduling in each one.
More info about the Team Build component can be found here.
8. Make it Easy for Anyone to Get the Latest Executable
Most builds produce useful output, such as an executable program, a packaged zip file, or other artifacts. Many people may need to get access to the latest executable to be able to run it or just to see what changed last week. Many times developers are not able to find it because there is not a well known place where these files are stored and they spend many hours just looking for this information.
As commented briefly in previous Practice 2 Automate the Build, the Ant build toolkit of RTC has a set of Ant tasks that can perform various operations on a build. Some of these tasks (artifactLinkPublisher and artifactFilePublisher tasks) enable your build to publish these artifacts. When using these tasks, the artifacts specified in your Ant builds are available in the build result editor, on the Downloads page, once the build finishes.
Figure 16.- Build Downloads Page
But, what happens when there is a failing build, or a problem in production with a specific version of the code? How can we recreate exactly the environment used in that version?
The RTC Build Team component creates a snapshot of the streams used in the build and such information is attached as part of the build results. The RTC Source Control component lets the developer to recreate a Repository Workspace from that snapshot. So just with a single click a developer can recreate that environment.
Figure 17.- Example of a Build Result with its snapshot
Figure 18.- Example of Repository Workspace creation from a snapshot
More info about the Team Build component can be found here.
9. Everyone can see what’s happening
Communication is critical to implement a good Continuous Integration system. Each team member needs to have easy and transparent access to the state of the system, last changes made and state of the mainline integration build. Visibility and transparency of the flow of information between team members are essential.
RTC implements a Dashboard component. Dashboards are a Web UI component intended to provide information about the project status at a glance. It provides for easy drill down to get more complete information. It also represents the integration point for the data provided by all RTC components.
Related to the Continuous Integration theme, the RTC Dashboard component includes several viewlets related to build activities, such as a history of executions with some basic information about whether it succeeded or failed, the date of the execution, etc… Of course, from each item the developer can navigate to the complete build result.
RTC also implements an event notification system so that each time a build is requested and has finalized, a notification or automatic email will be sent to all subscribers. These notifications can be automatically displayed to the user through the web interface or as a pop-up notification:
Figure 19.- Example of a Dashboard showing Build history and Event log viewlets
RTC can store history of past builds and developers can access to this data at any time to analyze trend results:
Figure 20.- Trend graph in build result
RTC also implements a Reporting component, so the development team can add to the Dashboard different reports related to the build durations, build health, build test errors, etc. Hence, at any point in the project’s life a development team will be aware of what is happening within the project.
Figure 21.- Example of reports related to the builds
Build features (build definitions, build results, build history and build engine status) are available in RTC through all clients (Web, Eclipse and Visual Studio):
Figure 22.- Access to build features through the web interface
More info about the Team Build component can be found here.
More info about the Dashboard component can be found here.
More info about the Reporting component can be found here.
10. Automate Deployment
To implement a Continuous Integration system you need to implement multiple environments (development, integration, production) to run your build and tests and, if possible, automate as much deployment between these environments as possible because you may be doing these deployments several times each day.
As we commented before, the RTC Team Build component can be installed multiple times in multiple different environments and platforms, all of them connected and collaborating with the same RTC Server.
One of those environments could be the test environment for regular builds, but another one could be for production where the build also deploy the code. Of course, deploying tasks will be part of the build script itself, so it depends of the target deployment application to understand which commands or procedures will be executed.
More info about the Team Build component can be found here.
Summary
Continuous integration is a software development best practice that distributed teams use more and more as a way to mitigate integration problems. Using Rational Team Concert integrated components you will be able to implement easily and effectively a Continuous Integration system that will help your team to achieve better results in your development.
Rational Team Concert provides a unique source control repository that integrated with some build features will allow you to optimize source code integration and to increase transparency and automation of your build processes.