Managing variants of the source code for Android OS using Rational Team Concert
Last updated: December 20, 2012
Build basis: Rational Team Concert 4.0.1
Table of Contents
- Getting started
- Configuring the source control clients
- Creating a variant strategy
- Component strategy
- Sharing code
- Stream strategy
- Flowing fixes between different streams
- Setting up caching proxies
IntroductionWorking with a large source code base can be challenging: even more so, if you maintain many variations based upon an original standard. Tracking fixes and improvements across a wide variety of streams creates a significant challenge. Properly planning to create a consistent variant strategy for source control can save the development team significant time. This article describes how to use Jazz source control to help you plan and develop a strategy for managing software variants. A software variant as a particular configuration of features in a product line that is selectable through the configuration and selection of software subcomponents. We will illustrate a strategy for making sense of the complexity of variance using the Android operating system (OS) as an example. Android  is a popular open source platform for mobile devices led by Google, and has wide adoption. This article does not provide any guidance beyond working with the Android platform/OS, although you can use similar approaches to manage variants of Android apps.
PrerequisitesIf you plan to work with the Android source code base, you should have experience downloading  and building  the Android project. Many Linux packages must be installed to build the Android source tree, so you must be familiar with this environment. Android takes time to compile, so you should set expectations accordingly. Additionally, it is helpful to have a good understanding of Jazz source control concepts .
MotivationThe scenario is as follows: consider a participant in the Android project, who wants to provide two different classes of tablets, and four different classes of smartphones. The tablets have equivalent processors but different screen sizes and radio chipsets. Of the four smartphones, there are two different form factors that are used. Within comparable form factors, there are different options as to what hardware components are available. Each type of hardware component may require custom drivers, and ideally, you want to reuse individual pieces of software where-ever they can apply. The more choices that manufacturers provide for their customers, the greater the opportunity for complexity in how the product gets built and fixed. While many of the aspects of variance are due to hardware or design choices, some are due to changing or emerging business strategies and pressures. In a development organization, teams must find ways to methodically organize and manage their different variants so that there is less development cost and risk when expanding or creating a new product line. Managing this complexity in a positive way is a competitive advantage.
Getting startedThe Android project has a bootstrapping tool  called repo which helps to download Android from numerous git repositories. Once you have successfully installed repo and git, you can use the following command to download the latest Android source code base.
Android has developed rapidly, and there are numerous versions that are available and supported by different handset providers and network operators as illustrated by the following reference. It is conceivable that any individual handset manufacturer will need to support many of these major versions of Android, and perhaps even more minor versions. Additionally, Google also works selectively in private branches , merging new changes into the public repositories periodically, so the lifecycle of the version branches at Android.com is really quite fluid.
$ repo init -u https://Android.googlesource.com/platform/manifest $ repo sync
Configuring the source control clientsWhen you share large code bases into Jazz source control, it is best to configure the source control clients so that they have adequate resources.
Increasing client memoryIncrease the client memory to avoid client-side OutOfMemoryErrors when dealing with large data sets and the general extra work that it requires to manipulate them. For the Eclipse client: in the ini file located as a sibling to the executable (e.g. eclipse.ini for installs that use eclipse.exe/eclipse to start), set the vm arg: -Xmx1024m. For the command-line client: edit the ini file that is a sibling to the executable (e.g. jazz/scmtools/eclipse/scm.ini) and set the vm arg: -Xmx1024m. For the lscm command line script/batch file, you will also want to add the option to the EXTRA_JAVA_OPTS variable declared in that script. For either client, you can additionally choose to set the minimum heap size setting. (-Xms) If users experience large pauses (from VM garbage collection), set the minimum heap setting close to or equal to the maximum heap setting. For example: -Xms1024m -Xmx1024m.
Increasing client timeoutIncreasing the client timeout value can be important if the source control streams contain large files (e.g. several gigabytes). If so, the upload of the file might timeout because the upload can be a very long running operation. To increase the Eclipse client timeout:
- In the Team Artifacts view, expand Repository Connections.
- Right-click on the repository; then click Properties.
- In the Properties window, in the left pane, select Jazz Repository Connection.
- In the Connection Timeout (in seconds) field, type 3600.
Increasing client threadsTo speed up large loads and shares, you can increase the client-side threadpool size. To increase client threads in the Eclipse client:
- Click Window > Preferences.
- In the Preferences window, expand Team and click Jazz source control.
- On the Jazz source control page, modify the Maximum number of threads field and click Apply.
Creating a variant strategyAfter the clients are properly configured, you should come up with a variant strategy to enumerate the number of variations within your product line. When dealing with software variants, think of categorizations of these many variants in terms of dimensions. One dimension that we have already been considering by default, is the version of Android. Each development branch of Android hosted at Android.com has its own lifecycle, and as a consumer, companies should expect to be required to sync in fixes and other changes from this as a source. Artifacts under Jazz source control are grouped into components. Any group of files and folders that share a common root directory can be a component. When trying to determine other dimensions, consider the different areas of source that the team expects to change consistently across the entire source tree. These areas can help define the granularity for Jazz source control components. Some pieces of the source code tree might apply to some configurations. Alternatively, there may be some subsets of the tree which are provided by an external party, such as a subcontractor or a business partner. These are areas that might be good candidates for creating a component for code isolation, and perhaps for consideration as an additional aspect of variance. Much of the Jazz source control security model is based upon protecting access to components and streams , and this approach has been used by many customers. Some of these components can change consistently together in many cases. Some can vary significantly across variants. In each of these cases, consider the differences to be additional dimensions in the overall variant strategy. Perhaps there are some dimensions that are more pervasive than others. Consider the diagram in Figure 1. Figure 1 can indicate a point in the variance space which can be a valid, supported version of the product. We have marked unnecessary variants with the black cross (X). In terms of a business strategy, it will likely not make sense for anyone to maintain streams for every version of Android compared to every chipset or display that exists in the organization’s supply chain. The universe of variants will likely contain some combinations which can not logically or practically exist. For this reason, there will be holes in the matrix, where not all values of each dimension are consistently satisfied. Consider the number of variants as being the upper limit of the cross product of all of the different dimensions. Using a simple example, if you supports N versions of Android, and M chipsets, and there are no other dimensions to consider, then the maximum number of variants that can be identified is NxM. This upper limit can serve as an indicator to help you set a component and stream strategy to support your product line.
Component strategyAs indicated in the previous section, choosing the right component boundaries to segment the source code is an important decision. Component boundaries provide isolation between areas of the code which change perhaps at different frequency or scope. There are other considerations, in terms of sizing. Jazz source control can deal with large components, but there is a practical recommendation, described in the RTC sizing guide  of 100,000 files per component. Therefore, you should structure your components so that they do not grow exceedingly large. Similarly, it is possible to break code down into components of too fine of a granularity. It can create some confusion in the RTC user interface if there are too many components. After downloading and building the Android source tree, we see over 420,000 resources in the tree. When unbuilt, the file tree has approximately 300,000 resources. For each top level folder, create a separate component and share it using the command line. The only exception is the “external” folder, as this folder is quite large. Break the external folder into four separate components. This creates the following list of components:
Sharing codeThe RTC command line provides a
sharecommand that allows for a directory to be shared into a component in a workspace. The example below describes how to share a single component (abi). You can reuse the pattern to share additional components.
Creating a repository workspaceCreate a repository workspace to perform the share in. For example:
The example above creates a repository workspace named “AndroidWS” on a server that I have already logged into and nicked named “kennyg”. The -e switch ensures that a default component is not created along with the workspace. An identifier “(2336)” which is displayed in the output. The identifier is an aliases that you can use as a shorthand when working interactively. The aliases in these examples are unique to the server that the examples were run against. In following these examples, you will need to substitute the values with the aliases that are produced from running your own commands.
$ lscm create workspace -e AndroidWS -r kennyg Workspace (2336) "AndroidWS" successfully created
Creating componentsCreate a component for each folder that was identified the previous section.
$ lscm create component -r kennyg abi 2336 Component (2337) "abi" successfully created. $ lscm list components 2336 Workspace: (2336) "AndroidWS" Component: (2337) "abi"
Sharing code into each componentSharing is to upload the source code into the RTC repository workspace. The example below shares the abi component to start.
Sharing code can take time: it is heavily dependent upon how well the RTC server has been tuned  and upon the speed of the network. Sharing the entire Android Open Source Project took us approximately 1.5 hours on a quad core RTC server with 8 Gigabytes of RAM. To validate that the file tree is in the workspace, run the following command:
$ lscm share 2336 2337 ./abi Shared successfully
When the sharing piece completes, the change set needs to be marked as completed. To see the change set alias for the complete command, run the status command:
$ lscm list remotefiles 2336 2337 --depth - / /abi/ /abi/cpp/ /abi/cpp/include/ /abi/cpp/include/cxxabi.h /abi/cpp/include/typeinfo /abi/cpp/include/new /abi/cpp/src/ /abi/cpp/src/pbase_type_info.cc /abi/cpp/src/pointer_to_member_type_info.cc /abi/cpp/src/new.cc /abi/cpp/src/delete.cc /abi/cpp/src/vmi_class_type_info.cc /abi/cpp/src/si_class_type_info.cc /abi/cpp/src/dynamic_cast.cc /abi/cpp/src/pointer_type_info.cc /abi/cpp/src/enum_type_info.cc /abi/cpp/src/type_info.cc /abi/cpp/src/array_type_info.cc /abi/cpp/src/function_type_info.cc /abi/cpp/src/fundamental_type_info.cc /abi/cpp/src/class_type_info.cc /abi/cpp/Android.mk /abi/cpp/.git/ /abi/cpp/.git/objects /abi/cpp/.git/logs /abi/cpp/.git/svn /abi/cpp/.git/refs /abi/cpp/.git/hooks /abi/cpp/.git/info /abi/cpp/.git/packed-refs /abi/cpp/.git/index /abi/cpp/.git/rr-cache /abi/cpp/.git/HEAD /abi/cpp/.git/description /abi/cpp/.git/config
Sharing the “external” folder is a more complicated than the “abi” examples. We want to break up the external folder into four separate components, with an approximately comparable number of folders shared. There are more than 150 external subfolders, so we should share approximately 35 to 40 folder per component. Share the first forty into external1, the next forty into external2, and so on. This can be scripted. Note: In order to share the external folders as a nested share, you can move the external folder out of your main Android loading space to an unrelated directory. For example if you loaded in $HOME/android, you can move external to $HOME/elsewhere. This will avoid an error where the command line indicates that the “ancestor is shared”. Once you have completed sharing all of the external folder source, you can move onto the next step, validating that the workspace loads and builds. At this point you should consider that the working area you originally ran repo in can now be replaced by a fresh workspace load, and you can abandon it to validate the next step.
$ lscm status Workspace: (2336) "AndroidWS" <-> (2336) "AndroidWS" Component: (2337) "abi" Baseline: (2339) 1 "Initial Baseline" Change sets: (2373) *--@ "Share" 11-Dec-2012 10:47 AM $ lscm cs complete 2373 Change set completed.
Loading the workspace back to diskTo test that the share completed successfully, load the repository workspace back to disk to ensure that everything loads as expected. Most of the components can load directly into the sandbox directory. Here is a pattern to load the abi component:
The external[1-4] components need to load in a special way. For these components, you will need to use a different syntax:
$ lscm load 2336 abi Successfully loaded items into the sandbox.
After all the code loaded, validate that the code builds and runs. Assuming that the code builds and runs successfully, you can create a load rule, that allows for a simple format for loading the entire sandbox in one command. The following command creates a loadrule in the loadrules directory in the users home directory.
$ mkdir external $ lscm load 2336 external4 -t external Successfully loaded items into the sandbox.
You can share this load rule by emailing it to team members, or by checking it into the repository. For details on using load rules, please see .
$ #run this in the sandbox root $ lscm create loadrules ~/loadrules Successfully created load rules.
Baselining the shared codeAfter you have validated that all of the code is shared and can be loaded, create a snapshot to capture this state.
$ lscm create snapshot --name "version_X" 2336 Snapshot (3422) "version_X" successfully created
Creating a stream and promoting the snapshotWith a snapshot that records the specific state, you can create a stream with this configuration.
To promote the snapshot to the stream so that team members can access it, run the following command:
$ lscm create stream --projectarea Android --snapshot 3422 "Latest Android Version" -r kennyg Stream (3423) "Latest Android Version" successfully created.
For more details about the Jazz source control share or create commands, see the documentation in the infocenter. If you want to script this process with the command line, use UUIDs instead of the numerical aliases that are used in the above examples. For more details, see . When sharing in a particular version (or tag) from Android, start with the earliest version possible. Sharing in the earliest release gives teams more flexibility in merging changes forward or backwards, because the history starts from common ancestry. Figure 2 illustrates the Android version dimension.
$ lscm snapshot promote 3423 3422 Promoted snapshots: 3422
- Start at the earliest version that the organization wants to support.
- Share in the code for this version.
- Create a snapshot on the stream to track this exact configuration. Promote the snapshot to an integration stream for safekeeping.
- If there is not a later or derived version to share, the sharing effort is complete.
- Overwrite the local environment now with the next major version of Android that the product supports. View outgoing (local) changes for all the components affected.
- Repeat step 2 with the more recent version of Android.
Stream strategyWhen it comes to stream strategies, there are different options. One option is to create a stream for every variant that is supported. In this case, it will likely be necessary to create a naming convention which allows the team to understand the dimensions indicated in the variants. A naming convention along the following lines could work:
Flowing fixes between different streamsAfter everything is shared, users can choose which stream to work against, depending on what their day to day tasks are. If a bug is fixed in a particular variant, you can port the changes to other variants. If the fix happened in an earlier (predecessor) variant, the changes often apply simply, with perhaps some merging required. In this case, you can track the bugfix easily using the Locate Change Sets traceability feature, which can greatly reduce the complexity of tracking and managing bugfixes. Alternatively, if the fix is very divergent from the target stream that it is to be applied to, you will likely need to create a textual patch. The patches support in Rational Team Concert is powerful but requires some discipline. It is important to have a consistent patching strategy to ensure that users, when they create patches, link appropriate work items to the change sets in order to maintain traceability. It is recommended that tool administrators experiment with the patching feature and create a process that they can communicate to their team as to how to consistently use work items and patch change sets together to properly manage the traceability.
Setting up caching proxiesIn order to speed up workspace loads and improve overall scalability of the SCM server, it could be worthwhile to setup a secure caching proxy configuration as outlined in this article . Caching proxies ensure that users that load the same content can retrieve it locally from a caching proxy server rather than going back to the main repository for every single request. Caching proxies provide signficant value when developers are working with large source code trees from multiple sites.
ConclusionRational Team Concert provides a modern source control system that can be used to successfully manage the complexity of dealing with a widely divergent code base. Jazz source control is continuing to evolve to improve this workflow.
- The Android Open Source Project @ Android.com
- Downloading the Source Tree @ Android.com
- Building and Running @ Android.com
- Getting Started with Jazz source control @ Jazz.net
- Version Control with Repo and Git @ Android.com
- Android version history @ wikipedia.org
- Branches and Releases @ Android.com
- The CLM sizing guide @ Jazz.net
- The SCM Command Line infocenter @ Jazz.net
- Controlling access to source control in Rational Team Control
- Tuning the Rational Team Concert 4.0 server
- Scripting using the Rational Team Concert SCM Command Line Interface
- Loading Content from a Jazz source control Repository in Rational Team Concert 4.0
- Using content caching proxies for Jazz source control
About the AuthorJohn Camelon is the technical lead for Rational Team Concert Source Control and serves on the Rational Team Concert Project Management Committee.
Copyright © 2012 IBM Corporation