Using Rational Team Concert for source code management in Jenkins jobs

Ralph Schoon, IBM | Kirk Woods, IBM | Tim Feeney, IBM
Sridevi Sangaiah, Persistent Systems | Lakshmi Narasimhan T V, Persistent Systems | Mehul Mehta, Persistent Systems
Last updated: 14 December 2020
Build basis: Rational Team Concert 6.0.6.1 and later, Build System Toolkit 6.0.6, Jenkins 2.150 and later, Rational Team Concert Plug-in 1.3.0 and later

This article provides guidance on using Rational Team Concert as a source code management system in Jenkins jobs. First, it introduces the build system in Rational Team Concert and provides guidance for loading source code from Rational Team Concert during a build, which are also applicable for Jenkins builds. Then, it introduces the integration of Jenkins with Rational Team Concert and provides guidance for configuring Rational Team Concert as a source code management system in Jenkins jobs.    

Contents

• Rational Team Concert build system
  • Build definition
  • Build workspace
  • Build engine
  • Build System Toolkit client
  • Build results
  • Personal builds
• Team Concert plug-in for Jenkins
  • Setting up the Team Concert plug-in for Jenkins
  • Jenkins Global Tool Configuration
  • Configuring the Team Concert plug-in
• Configuring Jenkins Jobs
  • General
  • Rational Team Concert source code management
  • Build triggers
  • Build environment
  • Build
  • Post-build actions
• Other considerations
  • Configuring WebSphere web containers
  • Dashboards
• Summary
• Related artifacts

Rational Team Concert build system

Rational Team Concert has its own integrated build management system. This section explains the basic concepts of the build management system and provides guidance for configuring the Rational Team Concert source control management (SCM) feature for loading content during a build. Most of this guidance applies when configuring Rational Team Concert as an SCM system in Jenkins jobs.

Build definition

Build definitions define what to build and how to build it. To request builds from within Rational Team Concert, a build definition is required. Each build definition must have a unique ID.

Build definitions define various options such as the build schedule, build properties to pass to the builds, and a policy for pruning build results. Rational Team Concert build definitions provide several pre-build and post-build options. By using the Jazz Source Control pre-build option, you can configure how the build uses the Rational Team Concert SCM system to load the files to build.

Build workspace

When you use the Jazz Source Control pre-build option, the build definition requires a unique repository workspace. This repository workspace loads files that are required in a build.

Repository workspaces can be created from a stream, another workspace, or a snapshot. Use repository workspaces that flow with a stream. Keep the workspace that you use for building separate from development workspaces. The build repository workspace must be owned by the user whose user ID is used for building.

Typically, automated build operations are run by user IDs, not people. User IDs, often called functional users, are created and maintained for this purpose. These user IDs are used in the automation to connect to Rational Team Concert and shared by the administrators for the build environment when needed for maintenance.

The build user must have access to the SCM data. To complete the required operations on the data in the Rational Team Concert SCM feature, this user must have a license that allows SCM operations. Rational Team Concert provides 50 Build System licenses that you can use for this purpose. The user must have permission to accept incoming changes and create snapshots. This often requires the user to be a member of project areas with roles required to complete the SCM and build operations.

Guidance

– Use a functional user as the build user when running the automation against Rational Team Concert.

– Use repository workspaces that flow with a stream.

Loading content

Source code must be loaded from the Rational Team Concert server to local storage before it can be worked on or built. To do this, use a Rational Team Concert client such as the Rational Team Concert Build System Toolkit, a Rational Team Concert IDE, shell integration, or the SCM command line.

Full load

When a configuration such as a repository workspace is loaded into an empty sandbox or load directory, all files must be received by the client. This scenario is called a full load.

Incremental load

The Rational Team Concert SCM feature keeps track of the files that are loaded into a sandbox. When loading a new configuration into an existing sandbox or load directory, the SCM feature detects which local file versions should be replaced, added, or removed. In this scenario, the Rational Team Concert client only needs to request file versions from the server that have changed or are unavailable. This scenario is called an incremental load.

In a typical incremental load scenario, a repository workspace is already loaded into a sandbox. During an incremental load, the Rational Team Concert client also deletes local files that are not part of the configuration. Local files that are filtered by the .jazzignore rules are not deleted.

For an incremental load, the number of files to request and load is typically very small when compared to a full load.

When each build has its own directory on the build machine, or the builds that share the same directory are loading workspaces based on the same stream (such as a personal build), the incremental load has an advantage: If two builds loading vastly different content share a build directory, the incremental build does not load much less than the full load.

For an incremental load, the Rational Team Concert build client completes more tasks than a full load, which can increase the build time.

Optimized incremental load

Optimized incremental load is a new method of load that is introduced in version 7.0.1. Optimized incremental load is similar to incremental load in that only modified files are fetched from the repository.

Optimized incremental load does not scan and revert local changes nor delete files that are not part of the configuration. It is suitable only in those builds that do not add, delete, or modify the loaded content. Optimized incremental load does fewer tasks than increment load, hence it takes less time when compared to incremental load or full load.

Optimized incremental load is not supported in the following scenarios where the build falls back to incremental load:

• Personal builds
• When you use dynamic load rules in Jenkins jobs

Caching proxy

Rational Team Concert supports caching proxies for SCM operations. When a Rational Team Concert client requests a file version, if it’s available on the caching proxy, it’s retrieved from there and returned to the client. If the file version is not available in the caching proxy, it’s retrieved from the Rational Team Concert server, stored in the caching proxy, and returned to the client.

When using a caching proxy, the server is only required to provide the file versions that are not in the cache. This removes much of the I/O and CPU load from the server. It also improves performance and reduces network traffic over a WAN when the client is remote and using a high-latency connection to the server.

The configuration of a caching proxy depends on the client. For the Rational Team Concert Build System Toolkit, provide the caching proxy URI instead of the Rational Team Concert server URI. In the Rational Team Concert Eclipse client, you must configure the network settings to use the caching proxy.

Consider using a caching proxy, such as Squid, to reduce the load on the server. With an effective caching proxy in place, there is little impact on the server between incremental and full loads after the initial load of files.

Load options

When using the Jazz Source Control pre-build option, the load options in a build definition are required to specify how and what to load, and the location to load into.

To configure the load location, specify a load directory, which can be a relative location. Each build definition must have its own load directory separate from other build definitions.

To configure the build definition in Rational Team Concert for incremental load, in version prior to version 7.0.1, ensure that you clear the Delete directory before loading check box, as shown in the following image.

In version 7.0.1 and later, select the Do an incremental load option to do an incremental load.

To build the most recent code, in the Accept Options section, select the Accept latest changes before loading check box. If the build has a defined schedule, select the Build only if there are changes accepted check box. 

Rational Team Concert provides templates for build definitions, such as defining which Build tool to run. A special build definition template is available for Jenkins.

Guidance

– To prevent the proliferation of build definitions, do not auto-generate them. Ensure that each build definition has a well-defined purpose. Delete build definitions when you no longer need them.

– Enable and configure build result pruning for each build definition by selecting the Deletion allowed check box or by creating a release from the build result to preserve the data longer.

– Use a different load directory for each build definition.

– In version 7.0.1 and later, select the Do an optimized incremental load option to perform an optimized incremental load. Select this option if you don’t have a caching proxy and your build execution is designed to handle some of the files generated in the previous build and does not modify the loaded files. Note that in version 7.0.1, optimized incremental load is not supported when using load rules; support for load rules was added in version 7.0.2.

– In versions prior to 7.0.1, clear the Delete directory before loading check box to perform an incremental load. In version 7.0.1 and later, select the Do an incremental load option. Select this option if you don’t have a caching proxy and your build execution is designed to deal with the remaining files generated from the previous build.

– In versions prior to 7.0.1, select the Delete directory before loading check box to perform a full load. In version 7.0.1 and later, select the Do a full load option. Typically, you select this option when your build execution is not designed to handle some of the files generated in the previous build: For example, files that the Rational Team Concert SCM features ignores based on the “.jazzignore” rules and, as a result, remain in the load directory.

If you select this option, the Build System Toolkit must use a caching proxy to communicate with the Rational Team Concert server.

Build engine

The build engine identifies the separate build server or application that completes builds. Each engine has a unique ID. Typically, each build engine runs on a dedicated platform. Rational Team Concert provides several build engine definitions to integrate different build systems, including these examples:

• Jazz Build Engine, which supports the Jazz Build Engine build system in Rational Team Concert 
• Hudson/Jenkins Engine, which supports the integration to Jenkins

Build engines run the build that’s described in the build definition.

Build System Toolkit client

The Build System Toolkit client contains the Jazz Build Engine and the build toolkit. The build toolkit is a collection of utilities that communicate build progress and results to the Jazz repository.

The build toolkit is also used in the Rational Team Concert Jenkins integration to complete the required SCM- and build-related tasks.

Build results

In Rational Team Concert, each build that is started creates a build result, which connects the artifacts that contribute to the build and can be used for reporting. Build results can include links to test files, log files, and other information.

You can upload build result contributions to build results and publish links to build results. You should not upload large files to the build result. To reduce the amount of data stored in the database, you should purge build results periodically.

Although you can store build result contributions in Rational Team Concert, consider storing them in an external repository. 

Guidance

– Avoid uploading large build result contributions, such as large files (logs, binary files, and archives). Store large build result contributions in web publishing and artifact repositories, and provide a link to them on the build result.

– Purge build results periodically when results are no longer needed.

Personal builds

When you request builds from Rational Team Concert, you can choose a personal repository workspace to run the build on. Personal builds enable developers to build and test their changes before they deliver them from their personal repository workspaces. Personal builds do not create a snapshot.

The Jazz Build Engine provides a filtering mechanism to determine which build engines process personal builds and which ones do not.

Team Concert plug-in for Jenkins

Jenkins is an open source automation server that automates tasks, including tasks related to building code. By using the Team Concert plug-in for Jenkins, you can configure Jenkins jobs to use Rational Team Concert for source control and optionally report information back to Rational Team Concert builds. 

To learn more about this plug-in, see the following resources:

• Jenkins Team Concert Plugin on the Jenkins wiki
• Integrating Rational Team Concert with Jenkins on Jazz.net
• An overview of the different build configurations supported by the Team Concert Jenkins plug-in on Jazz.net

Setting up the Team Concert plug-in for Jenkins

To use the Rational Team Concert SCM feature in Jenkins jobs, you must configure Jenkins to work with the Team Concert plug-in. Many configuration options are available and some of them can influence the build performance and the Rational Team Concert server load. Consider using a caching proxy such as Squid instead of going directly to the Rational Team Concert server.

Jenkins Global Tool Configuration

After you install the Team Concert Plugin in Jenkins, open the Jenkins application and click Manage Jenkins > Global Tool Configuration. 

The Team Concert plug-in uses the build toolkit from the Rational Team Concert Build System Toolkit client. Install the latest version of the Build System Toolkit client supported for your Rational Team Concert server.

Guidance

– Use the Build System Toolkit that ships with the Rational Team Concert server. This version of the toolkit supports all the features of the server version. The Rational Team Concert server must translate calls from older API versions, which can increase server load.

-To avoid breaking builds on different slaves, install the same version of the Build System Toolkit across all masters and slaves.

Configuring the Team Concert plug-in

You must configure global settings in the Team Concert plug-in. In the Jenkins application, click Manage Jenkins > Configure System, and in the Rational Team Concert section, specify values as shown in the following image.

Consider the following suggestions for the settings:

• Build Toolkit: Select the most recent compatible build toolkit version.
• Server URI: Provide the URI of a caching proxy, not the Public URI of the Rational Team Concert server.
• User credentials: Provide the credentials of a functional user that has a Build System License with access to the project areas and enough permissions to completed the required tasks.

Guidance

– To reduce load on the Change and Configuration Management (CCM) application server and database, specify a caching proxy for the Server URI.

Configuring Jenkins jobs

A Jenkins job is the Jenkins equivalent to a build definition in Rational Team Concert. To load content from Rational Team Concert SCM feature, consider the guidance provided as you configure Jenkins jobs.

General

On the General tab, notice the parameters that are used in the Jenkins jobs. Use the buildResultUUID parameter when the integration between Rational Team Concert and Jenkins enables builds to be requested in both Rational Team Concert and Jenkins. The buildResultUUID parameter is added automatically when a Build Definition in Rational Team Concert is associated with a Jenkins job.

Rational Team Concert source code management

To use the Team Concert plug-in in Jenkins Jobs to complete Rational Team Concert SCM tasks, select Rational Team Concert as the source code management option, as shown in the following image.

Although you can override the global Rational Team Concert repository connection, doing so increases the maintenance effort and complexity of the build environment.

If you must override the global settings, ensure that you specify a caching proxy URI in the Server URI field and specify credentials that have adequate permissions.

Guidance

– To reduce the maintenance required and to make it easier to enforce the use of a caching proxy, use the RTC Connection settings configured in the Manage Jenkins Global Tool Configuration (see Jenkins Global Tool Configuration section above) to specify the RTC repository connection configuration in the Jenkins Master/Slave. Do not select the Override global RTC repository connection check box.

– To reduce the load on the CCM application server and database, specify a caching proxy in the Server URI field.

– Specify a functional user that is assigned a Build System license.

Build configuration

The build configuration that you choose specifies the Rational Team Concert SCM artifact that loads content in a build.

Four types of build configurations are available for a Jenkins job: Build Definition, Repository Workspace, Stream, and Snapshot.

The following table lists the key characteristics and considerations of each build configuration.

Build definition

By choosing the Build Definition option, you can use the Jazz source control pre-build configuration options of a Rational Team Concert build definition.

You must provide a build definition ID, as shown in the following image.

For details, see the guidance in the Build definition section.

By default, when a Jenkins build is started from Rational Team Concert, the Jenkins console log is automatically added to the build record. This does not happen if the build is requested from Jenkins. To provide visibility, builds started from Jenkins would have to publish the logs back to Rational Team Concert.

Repository workspace

When you have a dedicated build workspace that flows with a team’s integration stream and traceability in Rational Team Concert is not required, choose the Repository Workspace configuration.

Create a workspace in Rational Team Concert specifically for this build configuration. The owner of the workspace must be the functional user that you created for running builds. 

You can configure the Jenkins Build Job to perform SCM tasks for the build like the Jazz Source Control pre-build option in the RTC Build Definition. The same guidance applies.

Similar to the Build Definition configuration, select the Delete directory before loading check box for a full load. If you are not using a caching proxy, do not select this check box.

Select the Accept latest changes before loading check box to pick up the changes that were delivered to the stream since the time the workspace loaded.

When you select the Accept latest changes before loading check box, Related Artifacts links are created for work items that are associated to accepted changes. The Related Artifacts links connect the work item and the build result in Jenkins. In version 1.2.0.5 of the Team Concert plug-in, you cannot disable this link creation.

Stream

When a build is based on a stream, a workspace is created for loading the files and a snapshot is created and the build runs. By using this configuration, changes are already in the stream and do no need to be accepted. Consider using this configuration for continuous validation of the integration state of a team’s stream across multiple jobs, without the need to create individual repository workspace for each job.

Snapshot

A build that is based on a snapshot is useful for re-creating a previous build, such as when a defect is found in production code that must be debugged and patched. This type of build would be triggered manually when needed.

You can also use this configuration to load and build consistent SCM data across concurrent runs of different or similar jobs, for example, concurrent builds of the same code for different platforms. Typically, an upstream job that is configured with a Build Definition, Repository Workspace, or a Stream creates a snapshot and triggers multiple downstream jobs that pass the snapshot details.

The load options of the Jazz Source Control pre-build configuration in a Rational Team Concert build definition also apply to the Jenkins jobs that are configured with a repository workspace, stream, or snapshot. The following guidance for loading content in a Rational Team Concert build also applies to Jenkins job configurations.

Guidance

– Use a different load directory for each Jenkins job.

– For an incremental load, clear the Delete directory before loading check box. Select this option when you don’t have a caching proxy and the build execution is designed to deal with the remaining files that were generated from the previous build.

– For a full load, select the Delete directory before loading check box. Typically, you select this option when a build execution is not designed for dealing with some of the files generated in the previous build, which are ignored by Rational Team Concert SCM depending on the .jazzignore rules, and remain in the load directory. Selecting this option automatically deletes all files, which requires the Build System Toolkit to communicate with the Rational Team Concert server through a caching proxy.

Build triggers

Depending on the build configuration that you choose, builds can be started from Rational Team Concert, Jenkins, or both.

To automatically start the build in Jenkins, configure the build trigger to poll SCM for changes. This configuration runs a build only when a change is detected.  The polling frequency should be no less than 30 minutes. Polling compares workspaces to determine if the build should start. If the polling frequency is too high, workspace comparisons cause increased load on the server.

If you select the Build periodically check box, a build is always scheduled whether or not changes are detected.

Guidance

– Select the Poll SCM check box and set a polling schedule of no less than 30 minutes to only build when needed. Setting a longer period between polls causes fewer workspace comparisons. For details, see Known Resource-intensive Scenarios.

– To limit the number of parallel compare workspace operations, set the Maximum limit of SCM workspace compare operations property in the advanced server properties of the CCM server. Specify a value equal to 25% of the number of CPUs.

Build environment

The configuration of the build environment depends on the plug-ins that are installed.

The Jenkins Workspace Cleanup Plugin provides an option to delete the workspace, including the load directory, before the build runs.

If the build is to complete an incremental load, clear the Delete workspace before build starts check box.

Guidance

– To complete an incremental load, clear the Delete workspace before build starts check box.

Other plug-ins, such as the Jenkins Timestamper plug-in enhance logging by enabling you to add timestamps to the Jenkins console output. With this enhanced logging, you can observe how long build steps take to complete.

When enabled, the build console in Jenkins shows timestamps that help you to determine the duration of build steps.

Build

Run the steps required to complete the build.

Post-build actions

Based on your project needs, you can select multiple post-build actions or several of the same actions from the following menu. 

Guidance

– Ensure that the RTC Post Build Deliver action is invoked only after the other build actions. You can configure this action to run only if a build is successful. However, configuring it to run before all the build actions finish might result in the system delivering unstable changes to the target stream.

– Invoke post-build actions only once. After this step runs, additional post-build deliver actions are ignored.

Other considerations

Configuring WebSphere web containers

When the Rational Team Concert server (running on WebSphere Application Server) loads many files or a number of large files into a workspace, it can run out of native memory. To prevent this problem, load the workspace one component at a time by using scripts. You can run the scripts as an unscheduled build so that you can repeat it as needed. After the workspace loads, it can be used for builds. Alternatively, you can set the WebContainer thread pool in WebSphere to use synchronous writes, which uses less memory but decreases performance. For details, see Setting the WebSphere Application Server WebContainer to synchronous mode at IBM Support.

Dashboards

By using the Team Concert Build system, you can add reporting and tracking information to the team dashboard. By integrating IBM Rational Quality Manager with the build, you can also track the validation of the build, which provides full traceability.

Summary

Using Rational Team Concert for source code management in Jenkins jobs provides the flexibility organizations and users need to produce software. Each option provides characteristics that can affect the performance of the Rational Team Concert server, the Jenkins build servers, or both. To make the best decision for a given build configuration, ensure that you are familiar with the characteristics of the options.

Related information

• Jenkins Team Concert Plugin
• Integrating Rational Team Concert with Jenkins
• An overview of the different build configurations supported by the Team Concert Jenkins plug-in
• Build Artifacts Publishing and Automated Build Output Management Using the Plain Java Client Libraries
• Separate personal and public builds on different build engines

About the authors

Ralph Schoon is a member of the Development Response Team, which has a mission to help IBM staff and customers apply Jazz technologies. Ralph has his own blog about extending and automating Rational Team Concert. He has published multiple articles on Jazz.net and is committed to working with Jazz users on the Jazz.net forums to help them understand the technical details of the Jazz products and optimize their solutions. Before his current role, Ralph worked as a Leading Technical Sales Representative for IBM, team lead and developer for machine control for an industrial company, developer, Microsoft Certified Professional, and system and network administrator. He can be contacted at ralph.schoon@de.ibm.com.

Kirk Woods is a Senior Consultant in IBM Engineering Lab Services. Kirk has more than 25 years experience as a software engineer/architect developing operating systems, firmware, commercial applications, and IT solutions. He has been consulting on the IBM IoT Continuous Engineering Solution for over 10 years. Kirk can be contacted at kmwoods@us.ibm.com.

Tim Feeney is a Senior Solution Architect who works with IBM’s key customers to architect lifecycle solutions, deployment topologies and usage models across the IBM IoT Continuous Engineering Solution; he uses these experiences to develop best practices and guidance and works with IBM Offering Management and Development to further improve the solution. Tim can be contacted at tfeeney@us.ibm.com.

Sridevi is a senior member of the Rational Team Concert Build team. She can be reached at sridevi_sangaiah@persistent.com.

Lakshmi Narasimhan is a senior member of the Rational Team Concert Build team. He can be reached at lnrasim7@in.ibm.com.

Mehul Mehta leads Rational Team Concert product development as Program Director and Chief Architect. Mehul has nearly two decades of industry experience, with many years spent on building products in the area of Change and Configuration Management. He can be reached at mehul_mehta@in.ibm.com.

© Copyright IBM Corporation 2019