Managing variants of the source code for Android OS using Rational Team Concert

John Camelon
Last updated: December 20, 2012
Build basis: Rational Team Concert 4.0.1

Table of Contents

1. Introduction
2. Prerequisites
3. Motivation
4. Getting started
5. Configuring the source control clients
6. Creating a variant strategy
7. Component strategy
8. Sharing code
9. Stream strategy
10. Flowing fixes between different streams
11. Setting up caching proxies
12. References

Introduction

Working with a large source code base can be challenging: even more so, if you maintain many variations based upon an original standard. Tracking fixes and improvements across a wide variety of streams creates a significant challenge. Properly planning to create a consistent variant strategy for source control can save the development team significant time.

This article describes how to use Jazz source control to help you plan and develop a strategy for managing software variants. A software variant as a particular configuration of features in a product line that is selectable through the configuration and selection of software subcomponents. We will illustrate a strategy for making sense of the complexity of variance using the Android operating system (OS) as an example. Android [1] is a popular open source platform for mobile devices led by Google, and has wide adoption. This article does not provide any guidance beyond working with the Android platform/OS, although you can use similar approaches to manage variants of Android apps.

Prerequisites

If you plan to work with the Android source code base, you should have experience downloading [2] and building [3] the Android project. Many Linux packages must be installed to build the Android source tree, so you must be familiar with this environment. Android takes time to compile, so you should set expectations accordingly. Additionally, it is helpful to have a good understanding of Jazz source control concepts [4].

Motivation

The scenario is as follows: consider a participant in the Android project, who wants to provide two different classes of tablets, and four different classes of smartphones. The tablets have equivalent processors but different screen sizes and radio chipsets. Of the four smartphones, there are two different form factors that are used. Within comparable form factors, there are different options as to what hardware components are available. Each type of hardware component may require custom drivers, and ideally, you want to reuse individual pieces of software where-ever they can apply. The more choices that manufacturers provide for their customers, the greater the opportunity for complexity in how the product gets built and fixed.

While many of the aspects of variance are due to hardware or design choices, some are due to changing or emerging business strategies and pressures. In a development organization, teams must find ways to methodically organize and manage their different variants so that there is less development cost and risk when expanding or creating a new product line. Managing this complexity in a positive way is a competitive advantage.

Getting started

The Android project has a bootstrapping tool [5] called repo which helps to download Android from numerous git repositories. Once you have successfully installed repo and git, you can use the following command to download the latest Android source code base. Android has developed rapidly, and there are numerous versions that are available and supported by different handset providers and network operators as illustrated by the following reference[6]. It is conceivable that any individual handset manufacturer will need to support many of these major versions of Android, and perhaps even more minor versions. Additionally, Google also works selectively in private branches [7], merging new changes into the public repositories periodically, so the lifecycle of the version branches at Android.com is really quite fluid.

Configuring the source control clients

When you share large code bases into Jazz source control, it is best to configure the source control clients so that they have adequate resources.

Increasing client memory

Increase the client memory to avoid client-side OutOfMemoryErrors when dealing with large data sets and the general extra work that it requires to manipulate them.

For the Eclipse client: in the ini file located as a sibling to the executable (e.g. eclipse.ini for installs that use eclipse.exe/eclipse to start), set the vm arg: -Xmx1024m.

For the command-line client: edit the ini file that is a sibling to the executable (e.g. jazz/scmtools/eclipse/scm.ini) and set the vm arg: -Xmx1024m. For the lscm command line script/batch file, you will also want to add the option to the EXTRA_JAVA_OPTS variable declared in that script.

For either client, you can additionally choose to set the minimum heap size setting. (-Xms) If users experience large pauses (from VM garbage collection), set the minimum heap setting close to or equal to the maximum heap setting. For example: -Xms1024m -Xmx1024m.

Increasing client timeout

Increasing the client timeout value can be important if the source control streams contain large files (e.g. several gigabytes). If so, the upload of the file might timeout because the upload can be a very long running operation.

To increase the Eclipse client timeout:

1. In the Team Artifacts view, expand Repository Connections.
2. Right-click on the repository; then click Properties.
3. In the Properties window, in the left pane, select Jazz Repository Connection.
4. In the Connection Timeout (in seconds) field, type 3600.

To increase the command-line client timeout: Edit the properties file: $HOME/.jazz-scm/preferences.properties and change the timeout key/value pair: repository.timeout: 3600

Increasing client threads

To speed up large loads and shares, you can increase the client-side threadpool size.

To increase client threads in the Eclipse client:

1. Click Window > Preferences.
2. In the Preferences window, expand Team and click Jazz source control.
3. On the Jazz source control page, modify the Maximum number of threads field and click Apply.

To increase client threads in the command-line client: Modify the content.threads property in $HOME/.jazz-scm/preferences.properties. Increase to greater than 10 (default). We do not recommend setting the value beyond 25 threads.

Creating a variant strategy

After the clients are properly configured, you should come up with a variant strategy to enumerate the number of variations within your product line. When dealing with software variants, think of categorizations of these many variants in terms of dimensions. One dimension that we have already been considering by default, is the version of Android. Each development branch of Android hosted at Android.com has its own lifecycle, and as a consumer, companies should expect to be required to sync in fixes and other changes from this as a source.

Artifacts under Jazz source control are grouped into components. Any group of files and folders that share a common root directory can be a component. When trying to determine other dimensions, consider the different areas of source that the team expects to change consistently across the entire source tree. These areas can help define the granularity for Jazz source control components.

Some pieces of the source code tree might apply to some configurations. Alternatively, there may be some subsets of the tree which are provided by an external party, such as a subcontractor or a business partner. These are areas that might be good candidates for creating a component for code isolation, and perhaps for consideration as an additional aspect of variance. Much of the Jazz source control security model is based upon protecting access to components and streams [10], and this approach has been used by many customers.

Some of these components can change consistently together in many cases. Some can vary significantly across variants. In each of these cases, consider the differences to be additional dimensions in the overall variant strategy. Perhaps there are some dimensions that are more pervasive than others. Consider the diagram in Figure 1.

Each box on the grid of Figure 1 can indicate a point in the variance space which can be a valid, supported version of the product. We have marked unnecessary variants with the black cross (X). In terms of a business strategy, it will likely not make sense for anyone to maintain streams for every version of Android compared to every chipset or display that exists in the organization’s supply chain. The universe of variants will likely contain some combinations which can not logically or practically exist. For this reason, there will be holes in the matrix, where not all values of each dimension are consistently satisfied.

Consider the number of variants as being the upper limit of the cross product of all of the different dimensions. Using a simple example, if you supports N versions of Android, and M chipsets, and there are no other dimensions to consider, then the maximum number of variants that can be identified is NxM. This upper limit can serve as an indicator to help you set a component and stream strategy to support your product line.

Component strategy

As indicated in the previous section, choosing the right component boundaries to segment the source code is an important decision. Component boundaries provide isolation between areas of the code which change perhaps at different frequency or scope.

There are other considerations, in terms of sizing. Jazz source control can deal with large components, but there is a practical recommendation, described in the RTC sizing guide [8] of 100,000 files per component. Therefore, you should structure your components so that they do not grow exceedingly large. Similarly, it is possible to break code down into components of too fine of a granularity. It can create some confusion in the RTC user interface if there are too many components.

After downloading and building the Android source tree, we see over 420,000 resources in the tree. When unbuilt, the file tree has approximately 300,000 resources. For each top level folder, create a separate component and share it using the command line. The only exception is the “external” folder, as this folder is quite large. Break the external folder into four separate components. This creates the following list of components:

• abi
• bionic
• bootable
• build
• cts
• dalvik
• development
• device
• docs
• external1
• external2
• external3
• external4
• frameworks
• gdk
• hardware
• libcore
• libnativehelper
• ndk
• packages
• pdk
• prebuilts
• sdk
• system
• tools

Also create a component for the root Makefile and call it Main.

Components are expected to live in multiple streams, and can serve similar or equivalent purpose in many different streams. In this case, the tools component likely can exist in multiple versions of Android, so for variant streams based upon those versions, we would expect them all to have a common component mapping to the logical tools repository in the Android repository. Some new RTC users make the mistake of trying to associate a component to a version number or timeline (e.g. tools_v3.0 and tools_v3.5 as separate components), which limits the team’s ability to flow changes between them.

Now that the component boundaries are set, share the code into Jazz source control.

Sharing code

The RTC command line provides a share command that allows for a directory to be shared into a component in a workspace. The example below describes how to share a single component (abi). You can reuse the pattern to share additional components.

Creating a repository workspace

Create a repository workspace to perform the share in. For example: The example above creates a repository workspace named “AndroidWS” on a server that I have already logged into and nicked named “kennyg”. The -e switch ensures that a default component is not created along with the workspace.

An identifier “(2336)” which is displayed in the output. The identifier is an aliases that you can use as a shorthand when working interactively. The aliases in these examples are unique to the server that the examples were run against. In following these examples, you will need to substitute the values with the aliases that are produced from running your own commands.

Creating components

Create a component for each folder that was identified the previous section.

Sharing code into each component

Sharing is to upload the source code into the RTC repository workspace. The example below shares the abi component to start. Sharing code can take time: it is heavily dependent upon how well the RTC server has been tuned [11] and upon the speed of the network. Sharing the entire Android Open Source Project took us approximately 1.5 hours on a quad core RTC server with 8 Gigabytes of RAM.

To validate that the file tree is in the workspace, run the following command:

When the sharing piece completes, the change set needs to be marked as completed. To see the change set alias for the complete command, run the status command: Sharing the “external” folder is a more complicated than the “abi” examples. We want to break up the external folder into four separate components, with an approximately comparable number of folders shared. There are more than 150 external subfolders, so we should share approximately 35 to 40 folder per component. Share the first forty into external1, the next forty into external2, and so on. This can be scripted.

Note: In order to share the external folders as a nested share, you can move the external folder out of your main Android loading space to an unrelated directory. For example if you loaded in $HOME/android, you can move external to $HOME/elsewhere. This will avoid an error where the command line indicates that the “ancestor is shared”.

Once you have completed sharing all of the external folder source, you can move onto the next step, validating that the workspace loads and builds. At this point you should consider that the working area you originally ran repo in can now be replaced by a fresh workspace load, and you can abandon it to validate the next step.

Loading the workspace back to disk

To test that the share completed successfully, load the repository workspace back to disk to ensure that everything loads as expected. Most of the components can load directly into the sandbox directory. Here is a pattern to load the abi component: The external[1-4] components need to load in a special way. For these components, you will need to use a different syntax: After all the code loaded, validate that the code builds and runs. Assuming that the code builds and runs successfully, you can create a load rule, that allows for a simple format for loading the entire sandbox in one command. The following command creates a loadrule in the loadrules directory in the users home directory. You can share this load rule by emailing it to team members, or by checking it into the repository. For details on using load rules, please see [13].

Baselining the shared code

After you have validated that all of the code is shared and can be loaded, create a snapshot to capture this state.

Creating a stream and promoting the snapshot

With a snapshot that records the specific state, you can create a stream with this configuration. To promote the snapshot to the stream so that team members can access it, run the following command:

For more details about the Jazz source control share or create commands, see the documentation in the infocenter[9]. If you want to script this process with the command line, use UUIDs instead of the numerical aliases that are used in the above examples. For more details, see [12].

When sharing in a particular version (or tag) from Android, start with the earliest version possible. Sharing in the earliest release gives teams more flexibility in merging changes forward or backwards, because the history starts from common ancestry. Figure 2 illustrates the Android version dimension.

The recipe for a series of successful shares would be as follows:

1. Start at the earliest version that the organization wants to support.
2. Share in the code for this version.
3. Create a snapshot on the stream to track this exact configuration. Promote the snapshot to an integration stream for safekeeping.
4. If there is not a later or derived version to share, the sharing effort is complete.
5. Overwrite the local environment now with the next major version of Android that the product supports. View outgoing (local) changes for all the components affected.
6. Repeat step 2 with the more recent version of Android.

Stream strategy

When it comes to stream strategies, there are different options. One option is to create a stream for every variant that is supported. In this case, it will likely be necessary to create a naming convention which allows the team to understand the dimensions indicated in the variants. A naming convention along the following lines could work:

• Jellybean_dimension1v1_dimension2v1
• Jellybean_dimension1v1_dimension2v2
• Jellybean_dimension1v2_dimension2v1
• Jellybean_dimension1v2_dimension2v2
• IceCreamSandwich_dimension1v1_dimension2v1
• IceCreamSandwich_dimension1v2_dimension2v1
• …

The benefit of this approach is that a target stream exists for every variant, and you can have build definitions for each stream to produce each variant. The drawback is that the number of streams can be quite large. If you are using the Eclipse client, for easier navigation of the streams, you can create a folder structure in the Team Artifact view Favorites section which allows you to categorize these streams into a directory structure of your choice.

Another alternative stream strategy is to house stream which are specific to the way that a subset of components vary, and then use build definitions to draw components from multiple streams. The benefit of this approach is that less streams are necessary. The build definition can take components external1 and external2 from stream Jellybean_external12_dimension1v1. The drawback is that this approach is less explicit: release engineers would need to update the build workspace when the dependencies change, and that policy change is not as visible to the overall team.

The strategy that an organization can take depends upon the team preferences and the magnitude of the variants.

Flowing fixes between different streams

After everything is shared, users can choose which stream to work against, depending on what their day to day tasks are. If a bug is fixed in a particular variant, you can port the changes to other variants. If the fix happened in an earlier (predecessor) variant, the changes often apply simply, with perhaps some merging required. In this case, you can track the bugfix easily using the Locate Change Sets traceability feature, which can greatly reduce the complexity of tracking and managing bugfixes.

Alternatively, if the fix is very divergent from the target stream that it is to be applied to, you will likely need to create a textual patch. The patches support in Rational Team Concert is powerful but requires some discipline. It is important to have a consistent patching strategy to ensure that users, when they create patches, link appropriate work items to the change sets in order to maintain traceability. It is recommended that tool administrators experiment with the patching feature and create a process that they can communicate to their team as to how to consistently use work items and patch change sets together to properly manage the traceability.

Setting up caching proxies

In order to speed up workspace loads and improve overall scalability of the SCM server, it could be worthwhile to setup a secure caching proxy configuration as outlined in this article [14]. Caching proxies ensure that users that load the same content can retrieve it locally from a caching proxy server rather than going back to the main repository for every single request. Caching proxies provide signficant value when developers are working with large source code trees from multiple sites.

Conclusion

Rational Team Concert provides a modern source control system that can be used to successfully manage the complexity of dealing with a widely divergent code base. Jazz source control is continuing to evolve to improve this workflow.

Reference links

1. The Android Open Source Project @ Android.com
2. Downloading the Source Tree @ Android.com
3. Building and Running @ Android.com
4. Getting Started with Jazz source control @ Jazz.net
5. Version Control with Repo and Git @ Android.com
6. Android version history @ wikipedia.org
7. Branches and Releases @ Android.com
8. The CLM sizing guide @ Jazz.net
9. The SCM Command Line infocenter @ Jazz.net
10. Controlling access to source control in Rational Team Control
11. Tuning the Rational Team Concert 4.0 server
12. Scripting using the Rational Team Concert SCM Command Line Interface
13. Loading Content from a Jazz source control Repository in Rational Team Concert 4.0
14. Using content caching proxies for Jazz source control

About the Author

John Camelon is the technical lead for Rational Team Concert Source Control and serves on the Rational Team Concert Project Management Committee.

Copyright © 2012 IBM Corporation