Jazz Library Integrating other SCM Systems with Rational Team Concert 2.0
Author name

workflow-management,rational-team-concert

Tue, 23 Jun 2009
5 min read
0

Integrating other SCM Systems with Rational Team Concert 2.0

David Lafreniere and Michael Valenta, SCM Team
Last updated: June 24, 2009
Build basis: Rational Team Concert 2.0

In this article, we will discuss several ways other Source Control Management (SCM) systems can be used in conjunction with Rational Team Concert (RTC) 2.0. We will be using Git, a relatively new and popular open source SCM in order to illustrate some of the points we make in our discussion. We will also provide source code for some of the integration points that we implement.

Table of Contents

Introduction

Although RTC comes with its own Source Control tool, there have been many requests for interoperability between other SCM systems and RTC work items. There are 3 levels of integration that we are going to discuss in this article:

  1. Integration within the RTC client: By this we mean using the Eclipse based client for your SCM of choice within the RTC rich client.
  2. Using RTC for in-house development for extensions to code from other SCM systems: In other words, how to manage a project that uses RTC for its development but is built on code that is contained in another repository type.
  3. Link RTC work items to changes in other SCM systems: We refer to this as providing a bridge between RTC work items and an SCM system.

In the following sections, we discuss each of these in more detail.

Integration within the RTC client

Integration with Eclipse is available for many SCM systems including CVS, Subversion, ClearCase and many others. There is a client for Git, called eGit, which has started to be developed at Eclipse.org and is available for download but has not been officially released yet. Since the RTC rich client is built on top of Eclipse, it is possible to install the Eclipse based clients for other SCM systems into RTC and use them along side of RTC work items, agile planning, build, or even Jazz SCM.

This level of integration is traditionally what has existed for many years between SCM, build, and work item systems. As with other such systems, a higher level of integration can often be achieved using scripts which we will look at more in depth in the section Provide an RTC to SCM Bridge.

Use RTC for In-House Development

In this section, we are going to assume that you are developing a product that incorporates one or more open source projects. When including open source projects in a product, it is common to make patches to the underlying open source code in order to fix bugs that you have encountered or to add new features that you may require. These patches are then contributed back to the open source project in the hope that they will be incorporated into a future release of the project. Because of this, the inclination is to use the same repository tooling for your project as is used for the open source product you are building upon. However, this is not a necessity and in this section we are going to give a simple example of how to use Jazz SCM to manage both your product code and the underlying open source code in a manner that allows you to contribute patches back to the open source project.

Patching Vendor Code Example

We are going to assume that you are building a product on top of an open source project that maintains its code in GIT. The article Managing Vendor/Third-party Code in Jazz SCM describes how you can use Jazz SCM to work with and manage source code that comes from a third party, whether the third party is a vendor or an open source project. We are going to extend what is presented in the article to describe how you could deliver those patches back to the open source project. We’ll use Git as the example repository type that the vendor code comes from, but the discussion could apply to CVS, Subversion or any other repository type as well.

Let’s assume that we have the code from an open source Git repository in our “Sweet Reporter” vendor stream and have patches in our development stream that we would like to contribute back to the open source project. The first step is to create a patch for the changes in the change set. This can be done by simply selecting New > Patch from the context menu of the change set in the Pending Changes view. In the screen shot below, we have changed our flow target to be the “Sweet Reporter” stream so we can see all of the modifications we have made in our development stream to the code from the vendor stream.


Creating a Patch From a Change Set
Figure 1: Creating a Patch From a Change Set


In the Create Patch wizard, all we need to do is specify a file name for the patch and click OK.

Create Patch Wizard
Figure 2: Create Patch Wizard


We can then submit this patch to the open source project through the appropriate channels. The patch file generated by RTC is formatted in the standard patch file format with some additional information which makes applying the patch in RTC easier. However, an open source project based on Git would most likely just use the standard “patch” command:

  patch -p0 < fix.patch

Because the patch format used is a standard format, other patching tools could be used, including the Apply Patch wizard in Eclipse or RTC.

Provide an RTC to SCM Bridge

One of the most useful features provided in RTC is the ability to link a work item to a change in the SCM system. The link integration between the work items and SCM components provided as part of RTC is very tight because both components are included in the RTC server and hence have a high degree of integration. However, it is possible to provide a similar type of integration between RTC work items and other SCM systems by using comments and the “Related Artifacts” links provided by the Work Items component. The level of integration will not be as high as with RTC SCM but it is useful none the less.

In this section, we are going to present an example that provides linking between RTC work items and Git changes. First, we will describe the REST API of RTC work items and how it can be used to add comments and “Related Artifacts” links. Then we will talk about Git and the various hooks it provides that are used to extend its functionality. Finally, we will discuss how we combine these two features to provide a basic level of integration between RTC work items and Git changes.

RTC REST API

RTC supports a simple RESTful API for modifying work items and their attributes. This is the key factor that allows simplified integration with other SCM systems. The original and experimental API introduced in RTC 1.0.1 can be visited here, while the current API for RTC 2.0 is found here. Although this article will only explore the API related to individual work items and their attached links and comments, the REST API supports the retrieval and modification of a wide range of other Jazz resources such as workflows, users, iterations, team areas, etc. As such, it is not surprising that this web API follows a resource-oriented architecture (ROA) in support of common CRUD functionality. When using ROA and CRUD in a web environment, the basic HTTP methods on resources — POST, GET, PUT, and DELETE — are generally taken to mean Create, Read, Update, and Delete.

In order to take full advantage of the REST API, an additional tool called cURL is often used. cURL is a command line tool that can perform URL manipulations and transfers with ease. This command line tool allows the transferring of files with URL syntax, supporting FTP, HTTP, HTTPS, TELNET… etc, as well as supporting SSL certificates, HTTP POST, HTTP PUT, cookies, user/password authentication, etc. CURL’s simple structure — a single cURL call with a few parameters — makes it the ideal tool to embed in automated scripts.

Similar to other websites, there must be some form of authentication established before the web service can be used. In our case, the process involves connecting to the RTC server, establishing a session, providing a valid username and password as well as the storing and sending of cookie data. This relatively complex task can be performed in cURL using the following two lines:

  $ curl -k -L -b <COOKIE_PATH> -c <COOKIE_PATH> <HOST>/authenticated/identity    $ curl -k -L -b <COOKIE_PATH> -c <COOKIE_PATH>    	-d j_username=<USERNAME> -d j_password=<PASSWORD> <HOST>/j_security_check

These steps will always be required when using the RTC REST API, thus it is important at this point to explore the specifics of each cURL parameter used. (Additional information for all of the cURL options can be seen on the cURL man page)

  • -k: This option explicitly allows cURL to perform “insecure” SSL connections and transfers. All SSL connections are attempted to be made secure by using the CA certificate bundle installed by default. This makes all connections considered “insecure” to fail unless -k/–insecure is used.
  • -L: If the server reports that the requested page has moved to a different location (indicated with a Location: header and a 3XX response code) this option will make cURL redo the request on the redirected URL.
  • -b: This points to a local cookie file. This allows the HTTP server to receive all the cookie data within the file.
  • -c: This specifies the file that cURL is allowed to write cookie data to after a completed operation. cURL writes all cookies previously read from a specified file as well as all cookies received from the remote server. The file will be written using the Netscape cookie file format. If you set the file name to a single dash, “-“, the cookies will be written to stdout.
  • -d <data>: Sends the specified data in a POST request to the HTTP server, in a way that is the same as if a user has filled in a HTML form and pressed the submit button. Note that the data is sent exactly as specified with no extra processing (with all newlines cut off). The data is expected to be “url-encoded”. This will cause cURL to pass the data to the server using the content-type application/x-www-form-urlencoded.
  • <COOKIE_PATH>: This is the path to the cookie file you wish to use. (ex: “/tmp/cookies.txt”)
  • <HOST> This is the URL path to your RTC server (ex: “https://My_RTC_Server_Path:9443/jazz”)

Once the two cURL lines have been run (and the authenticated session established), we can now freely use the Work Item REST API. In order to implement basic GIT and RTC integration, knowledge of the required API calls must be obtained. As previously mentioned, this article will explore the functionality related to retrieving a work item, creating a comment on a work item, as well as adding a “Related Artifacts” link.

Obtaining a work item in XML format is easily performed using the following line:

  $ curl -k -L -b <COOKIE_PATH> -c <COOKIE_PATH> <HOST>/oslc/workitems/<WORKITEM_NUM>.xml

The next two examples illustrate how one would add a “Related Artifact” link to a work item, as well as add a new comment. These examples both send JSON POST data to a particular work item’s “links” or “comments” collection. POSTs of this form generally require the knowledge of two things:

  1. The specific URL of the collection.
  2. The code snippet (POST Data), either in JSON or XML form, which describes the new attribute/resource

Command line cURL for adding a “Related Artifacts” link to a work item:

  $ curl -k -L -b <COOKIE_PATH> -c <COOKIE_PATH>   	-H "Accept: application/json" -H "Content-Type: application/json"   	-d '{"rdf:resource":"http://www.example.com","oslc_cm:label":"My Description"}'   	$HOST/oslc/workitems/<WORKITEM_NUM>/  	rtc_cm:com.ibm.team.workitem.linktype.relatedartifact.relatedArtifact

Command line cURL for adding a comment to a work item:

  $ curl -k -L -b <COOKIE_PATH> -c <COOKIE_PATH>   	-H "Accept: application/json" -H "Content-Type: application/json"    	-d '{"dc:description":"My new comment"}'   	<HOST>/oslc/workitems/<WORKITEM_NUM>/rtc_cm:comments

The noticeable difference with these commands is that we are now specifically modifying the header of the POST to allow the sending and receiving of the JSON MIME type (“application/json”). The “-H” supports the addition of new header information, or the overwrite of existing header information. When creating a new “Related Artifacts” link, the user simply has to modify the “resource” element to point to the desired URL, and modify the “label” element which shows up as the visible text in the work item. Likewise, for adding a new comment, the user simply has to add the desired text to the “description” element.

Git

Git is a free and open source, distributed version control system. Git was initially designed and developed by Linus Torvalds for Linux kernel development. One of Git’s strong points is that every Git working directory is a full-fledged repository with complete history and full revision tracking capabilities, not dependent on network access or a central server.

Getting started with Git is easy. For example, the following scenario describes the basic steps for creating a new repository with some initially committed files:

  $  cd <project directory>  $  git init  $  ... (create and modify some files)  $  git add .  $  git commit -m "WorkItem:1 Initial Commit"  $  git push <public repository location>

As we can see, the general source control management workflow when using Git mainly involves the commands “add” and “commit”. “Push” is also used to migrate these changes from a local/private repository to a remote/public repository. To further clarify how these commands are used, the following list explains specifically what each command does.

Commands:

  • git init – Create an empty Git repository or reinitialize an existing one
  • git add <filename> – Add file contents to the index (“.” will add all files in the directory)
  • git commit – Record changes to the repository
    • -m – To specify the comment string
    • -a – To commit all known files. (Same as performing “git add .” before committing)
  • git push – Update the remote repository to reflect the change history of the local repository

In order to integrate Git and RTC work items, we must be able to extend the functionality centered around the “commit” command. Another possibility is to ignore individual “private” commits, and add functionality centered around a Git “push” command. This would prevent private repositories from interfacing with RTC work items, while instead updating work items only when a collection of commits are “pushed” to a public Git repository. To extend Git’s functionality in such a manner, there are two approaches that could be taken. The first and more difficult approach would be to modify the actual Git source code to perform the desired functionality when a commit or push command is invoked. Since Git is open source this is certainly a feasible approach, however it would require that users continuously merge their changes with newer versions of Git releases. Fortunately there is another alternative which can provide the same RTC work item integration possibilities while at the same time eliminating a lot of the difficulty. This other approach involves the use of Git hooks.

Hooks are essentially executable files or scripts that can be run to extend the functionality of Git. In general, many of these hooks will run before, during and after certain “events”: most notably, “commit” and “push”. When “git init” is run (as in the example above), several example hooks are copied into the hooks directory (.git/hooks) of the new repository, but are disabled by default. To enable a hook, one simply has to rename it by removing its .sample suffix. The list of hooks available is shown in the figure below.

Available Git Hooks
Figure 3: Available Git Hooks


One important difference between Git hooks and hooks traditionally found in other revision control systems (CVS, Subversion, Perforce, etc.), is that Git hooks can be run locally as opposed to being exclusively server side. This approach allows individual users to customize how their own hooks behave, while still incorporating the idea of “server side” hooks by using them on “public” Git repositories. Clearly Git hooks could be used for a wide range of applications. For example, hooks could be used to prevent commits in which the source files do not compile, they could check files for correct format or spelling, or be used to send e-mails to specific developers with commit information.

While the next section “Git and RTC Integration” will explore the specifics of each hook in support of RTC work item integration, the following list will provide a general introduction to some of the hooks used.

commit-msg:
  • This hook is invoked by “git commit”.
  • This hook takes a single parameter which is the name of the file that contains the commit log message (comment).
  • If the script exits with a non-zero status, this will cause the “git commit” to abort.
  • This hook is allowed to modify the commit message file and as such it can be used to standardize the message to a predefined format
post-commit:
  • This hook is invoked by “git commit”.
  • It takes no parameters and is invoked after a commit is made.
  • This hook cannot affect the outcome of “git commit” and is primarily for notification.
update:
  • After running a “git push” on a local repository, this hook will be invoked by a “git-receive-pack” on a remote repository.
  • This hook will be run before updating the remote repository.
  • If the script exits with a non-zero status, this will cancel the update on the remote repository.
  • This hook takes three parameters:
    1. The name of the ref being updated (ex: “refs/heads/master”)
    2. The old revision object name stored in the ref
    3. The new revision object name to be stored in the ref
post-receive:
  • After running a “git push” on a local repository, this hook will be invoked by a “git receive-pack” on a remote repository.
  • This hook will execute on the remote repository only once, and after all the refs have been updated.
  • It takes no arguments directly, however it does receive the same information as the update hooks on its standard input.
  • Exiting this hook with a non-zero status will not abort any updates, and as such, similar to the post-commit hook, is used primarily for notification

Git and RTC Integration

After a brief introduction to some of the technologies used, we can now explore in depth how Git, its hooks, cURL and the Work Item REST API can collaborate together. Before exploring the details of each hook however, the following screen shots are presented in order to help gain a better vision of the end goal. Figure 4 below shows an example work item with no links attached. Figure 5 on the other hand shows a “Related Artifacts” link which was added from running a Git hook. It is also shown that this newly added link points to the Gitweb URL corresponding to the specific commit id.

Work Item Links - Before Running Git Hooks
Figure 4: Work Item Links – Before Running Git Hooks


Work Item Links - After Running Git Hooks
Figure 5: Work Item Links – After Running Git Hooks


The functionality shown above describes at a minimum what would be required to establish useful Git “commit” and RTC work item relationship. However, it is not only possible to link a work item to the specific commit ID on Gitweb, but using hooks, it is also possible to create bidirectional navigation so that each Gitweb “commit” page can link back to the related RTC work item page. The details of implementing this feature will be discussed shortly, however for now, Figure 6 below can demonstrate an example of this feature.

Gitweb Showing a Work Item Link
Figure 6: Gitweb Showing a Work Item Link


We have seen how Git can automatically create a “Related Artifacts” link on a work item, however from the work item alone, there is no way to view information related to the Git commit without navigating to the Gitweb page. It might be desirable for some users to have Git automatically display some “commit” related information by creating a new comment in the work item. This information may especially be useful in the case that Gitweb is unavailable which would render the “Related Artifacts” link useless. Figure 7 below shows an example of what a Git hook can do automatically when committing (or “pushing”) a new change set. It is also important to note that users are free to define their own custom comment “format”, as well as what specific information is displayed. For example, it is even possible to show in the work item comment the list of all the individual files that were changed in that specific Git commit.

Work Item Showing a Comment Added by a Git Hook
Figure 7: Work Item Showing a Comment Added by a Git Hook


As already mentioned, Git hooks are essentially scripts which run at certain points in time. In fact, any executable file, as long as it has the correct name, can be used as a hook. Because Git is predominantly used in a Linux environment, many of the existing hooks are written as a shell script, yet as expected, they can be written in nearly any language. For this article, we have chosen to write the various hooks using Perl. In order to take advantage of cURL in a Perl environment (as opposed to single command line cURL calls), we have made use of a cURL library (libcurl) for Perl called WWW:CURL:EASY.

Using WWW:CURL:EASY is not as simple as the name may imply, however it is possible to perform all the same actions that are available when using the normal command line cURL. Despite the added complexity, when using cURL in a script, one has full access to view and parse any of the headers, response body, return codes, etc, as well as perform additional actions depending on their values. One of the fundamental differences with libcurl used in scripts is that there is no simple means to add options (“-k, “-b”, etc.) as one would through command line. A complete list of all the available options can be seen here, while a list of the previously mentioned options and their equivalent libcurl options is shown below.

  -v (verbose)    <--> setopt(CURLOPT_VERBOSE, 1)  -k (insecure)   <--> setopt(CURLOPT_SSL_VERIFYPEER, 0) AND                       setopt(CURLOPT_SSL_VERIFYHOST, 0)  -L (location)   <--> setopt(CURLOPT_FOLLOWLOCATION, 1)  -b (cookie)     <--> setopt(CURLOPT_COOKIEFILE, $COOKIE_FILE)  -c (cookie-jar) <--> setopt(CURLOPT_COOKIEJAR, $COOKIE_FILE)  -d (post data)  <--> setopt(CURLOPT_POST, 1) AND                        setopt(CURLOPT_POSTFIELDS, $postData) AND                        setopt(CURLOPT_POSTFIELDSIZE, length($postData))  -H (header)     <--> setopt(CURLOPT_HEADER, 1) AND                       setopt(CURLOPT_HTTPHEADER, @HEADER)

For all “connections” to the RTC server, whether it be to send the login information for authentication, retrieve a work item, or to add a link or comment, the general Perl+cURL structure is all the same. This general form can be shown the simplified pseudo-code below.

  1. Set the cURL options  2. Define a variable/file to hold the response  3. "Execute" the cURL connection  4. Examine the cURL connection code  	If valid...  	4.1. Examine the HTTP return code  		If code = expected code (ex: 200)...  			4.1.1. Perform desired action (ex: continue script)  		If code = error code (ex: 404)...  			4.1.2. Perform desired action (ex: abort script)  	If not valid...  	4.2 Connection failed - Abort script

The below code is a concrete example of validating that a work item exists. The structure of this code is exactly the same as the pseudo-code above. As expected, a return code of 200 indicates that the work item has been returned successfully. However if the return code is 302 this indicates that a re-direct is required. In this particular case, the RTC server will re-direct users to a login page if they do not already have a valid session, and thus a warning to the user is shown stating that they are not authenticated. As you may notice, this particular script is used before a commit takes place in Git, and will abort the Git commit under all error conditions. 

  my $CURL = new WWW::Curl::Easy;    $FQHOST = $HOST."/oslc/workitems/".$WORKITEM.".xml";    # 1. SET THE CURL OPTIONS  $CURL->setopt(CURLOPT_VERBOSE, 1); # Prints extra information  $CURL->setopt(CURLOPT_SSL_VERIFYPEER, 0);  $CURL->setopt(CURLOPT_SSL_VERIFYHOST, 0);  $CURL->setopt(CURLOPT_COOKIEFILE, $COOKIE_FILE);  $CURL->setopt(CURLOPT_COOKIEJAR, $COOKIE_FILE);  $CURL->setopt(CURLOPT_HEADER,1);  $CURL->setopt(CURLOPT_URL, $FQHOST);    # 2. DEFINE THE VARIABLE TO HOLD THE REPONSE  my $response_body;  open (my $tempFile, ">", $response_body);  $CURL->setopt(CURLOPT_WRITEDATA, $tempFile);    # 3. EXECUTE THE CURL CONNECTION  my $retcode = $CURL->perform;    # 4. EXAMINE THE CURL CONNECTION CODE  if ($retcode == 0) {     print("Connection to Work Items server established...n");     my $response_code = $CURL->getinfo(CURLINFO_HTTP_CODE);          #print("Received response: $response_bodyn"); #print HTTP response (if needed)          # 4.1. EXAMINE THE HTTP RETURN CODE     if ($response_code == 200) {        print("Successfully verified that work item " . $WORKITEM . " exists.n");        exit 0;     } elsif ($response_code == 302) {        print("WARNING: Not Authenticated.n");        print("Aborting commit...nn");        exit 1;     } else {        print("WARNING: Work item $WORKITEM does not exist.n");        if ($ABORT_COMMIT_ON_INVALID_WORK_ITEM_NUM eq "true") {           print("Aborting commit...nn");           exit 1;     } else {           print("Continuing to commit...nn");           exit 0;     }     }  } else {     print("An error happened: ".$CURL->strerror($retcode)." ($retcode)n");     print("Aborting commit...nn");     exit 1;  }

The other important criteria for implementing a user friendly integration is to identify when and how a user is to associate a Git commit to a work item. The easiest approach is to define a specific string that users can specify in the commit comment which can be picked up in the hooks. For the provided hooks, the string “WorkItem:###” must be at the beginning of the comment string. Note however, the actual string can easily be modified as it appears as a “global” variable at the top of the scripts.

The following sections will explore each hook in more detail and identify the specific role they play in the overall Git and RTC Work Items integration. It is important to note that all of these hooks have important global variables that will have to be changed on an individual basis. These reflect individual usernames and passwords, as well as the specific HOST URL for the RTC server.

Example Script: commit-msg

The example script can be downloaded here. Please remember to remove the .sample extension to enable this example script.

This hook is intended to be used in a private repository. In short, after a “git commit” is called, this hook will check the comment file and attempt to find the work item search string at the start of the comment. It will ultimately decide to abort the commit or allow it to continue, depending on a variety of conditions.

As explained above, when using the RTC REST API an authenticated session must first be obtained from the server. Without going into to much detail of how this is done, it is enough to know that it accomplishes this task by first calling the two subroutines: REST_Authentication() and REST_j_security_check(). These subroutines create a session on the Server, send the username/password in a POST and store the cookie data in a file for future use. It is exactly as is shown for the command line cURL in the section RTC REST API above.

Before creating this authenticated session however, the script first checks to see if the comment actually has a work item reference (ex: “WorkItem:###”). The default behavior for this hook is to abort the commit if the work item search string is not present. Figure 8 below shows an example printout of a commit in which the user did not specify a work item in the comment.

Validation Checks - No Work Item String Exists
Figure 8: Validation Checks – No Work Item String Exists

It may be the case that a particular software team wants to ensure that every commit will be in relation to a known RTC work item. Enforcing this rule helps aid in project traceability, however it may be annoying if the user wants to add changes to his own personal files. This example hook can easily relax this requirement by simply changing the global variable called $REQUIRE_WORK_ITEM_COMMENT to “false”. Doing so will then allow normal commits which do not have any work item search string in them to proceed as normal. If there is a work item search string in the comment, or $REQUIRE_WORK_ITEM_COMMENT is set to “true”, then the hook will attempt to verify that the work item exists by retrieving the work item from the RTC server. Another check must be made at this point as well. It is possible for a user to reference a work item that does not exist. For example, the user may type “WorkItem:999999999” which obviously will not exist, or another possibility is that the work item was somehow removed from the RTC server. For integrity purposes, the default behavior is to abort the commit if the comment references a work item that does not exist. This behavior is demonstrated in Figure 9 below. Again however, we could allow user to reference “unknown” work items by setting the global variable $ABORT_COMMIT_ON_INVALID_WORK_ITEM_NUM to “false”.

Validation Checks - Invalid Work Item
Figure 9: Validation Checks – Invalid Work Item

The hook attempts to retrieve the work item by calling the subroutine REST_retrieve_work_item(). The layout for this subroutine was actually shown in the simplified example above. This is where it determines if it should continue with the commit or abort it, depending on the server return codes.

Another important feature of this hook is that it creates the means for bidirectional navigation between the RTC work item and the individual Gitweb page for the commit. Because this hook has access to the comment file for the commit, it can easily modify it as needed. There is a global variable called $APPEND_WORK_ITEM_URL_TO_COMMENT which is set to “true” by default. This causes the URL for the specific work item to be appended to the end of the comment on a new line. This is exactly why it shows up on the Gibweb as shown in Figure 6 above. Currently, this URL is only appended if the hook can successfully retrieve the specified work item, if it cannot the comment will remain untouched.

Example Script: update

The example script can be downloaded here. Please remember to remove the .sample extension to enable this example script.

This hook is intended to be used in a public repository. This script is almost identical in functionality to the commit-msg hook. The fundamental difference is that this hook is used to abort a “push” while the other one aborts a “commit”. Also, the commit-msg hook has to handle one commit, whereas the update hook is responsible for handling approvals for a collection of commits. The same permissions such as $REQUIRE_WORK_ITEM_COMMENT and $ABORT_PUSH_ON_INVALID_WORK_ITEM_NUM are all present in this hook as well, and each has the default value “true”. The difference in this case is that $REQUIRE_WORK_ITEM_COMMENT is now meaning that the work item search string must appear in the comment of EVERY commit in order to accept a push.

Since this hook must handle multiple commits, an additional subroutine is added called parseCommitComments(). This subroutine identifies each new commit that is trying to be added to the repository, and attempts to parse out the work item search string. If it is present, it will attempt to retrieve the corresponding work item, and use that as a basis to abort/approve the push.

The input parameters to this hook are also different from the commit-msg hook. In this case we are passed a reference to an object representing the “before” state, and the “after” state. In order to identify all the individual commits, we can use the built in Git function shown below by using the input parameters received by this hook.

  git-whatchanged $oldrev..$newrev

This command is useful, however it returns a wide range of information about each commit; whereas all we really need is the individual comments and the commit IDs. Git can help again with this problem by providing a formatting option for the ‘git-whatchanged’ function. The code below shows how we can easily obtain what we need.

  git-whatchanged $oldrev..$newrev      --pretty=format:"<COMMENT>%s<COMMENT> <COMMIT_ID>%H<COMMIT_ID>"`;

Using “%s” tells git to print the comment, while “%H” refers to the commit ID. We can also add any custom text in between any of the “%” references. Thus to make parsing easier in perl, we can add comment and commit ID delimiters before and after the %s and %H. This way if the comment spans multiple lines, it will still have the delimiters on either side which we can find easily using Perl.

Example Script: post-receive

The example script can be downloaded here. Please remember to remove the .sample extension to enable this example script.

This hook is also intended to be run on a public repository. It’s general purpose is to run after the update hook so that it can add the “Related Artifacts” links and comments to each work item. It is important to update the work item at this stage because this hook will only run after the push was a success. In other words, we need to be 100% sure that the new changes have been committed in the public repository before updating the work items with new comments or links.

For simplicity purposes, this hook does not perform the REST_Authentication() and REST_j_security_check() subroutines in an attempt to create an authenticated session. It does however pass in the cookie file which was created in the update hook. The assumption is that since this runs immediately after the update hook, the session should still be authenticated and not have expired. This is done simply because this code is intended to be for demonstration purposes only, but ideally one would include more in depth error checking for possible session/authentication problems.

This hook receives a “before” and “after” state similar to the update hook. However in this case we want additional information pertaining to each commit. This information can be used in the comment that is to be added to the work item. Thus, similar to above, we use this actual Perl code to allow easy parsing of the desired information:

  my $gitWhatChangedCommand = "git-whatchanged $oldrev..$newrev --pretty=format:" .  	""$commitIDDelimiter%H$commitIDDelimiter " .  	"$commentDelimiter%s$commentDelimiter " .  	"$commitDateDelimiter%ai$commitDateDelimiter " .  	"$authorDelimiter%an$authorDelimiter " .  	"$emailDelimiter%ae$emailDelimiter"";

Adding a comment for each new commit may not be a desirable feature for all teams, thus setting the global variable called $ADD_WORK_ITEM_COMMENT to “false” allows an easy means to disable the addition of a comment. Using scripts in this manner however, the user is free to choose any format of commenting they see fit. As a simple demonstration of this idea, the following global variables can easily be turned on or off depending on what information is required.

  # WORK ITEM COMMENT PREFERENCES  my $PRINT_COMMIT_DATE = "true";  my $PRINT_AUTHOR_NAME = "true";  my $PRINT_AUTHOR_EMAIL = "true";  my $PRINT_COMMENT = "true";  my $PRINT_COMMIT_ID = "true";  my $PRINT_GIT_URL = "true";

Once the desired comment text is gathered, adding the comment to the work item involves simply posting the JSON text to the comments collection as specified in the RTC REST API section.

This hook also adds the “Related Artifacts” link. Again, this simply involves posting the appropriate JSON fragment to the links collection. For the visible text of the link however, we have chosen to display the Git commit comment, but we have parsed out the work item search string at the beginning. It would be pointless to have “WorkItem:### …” displayed several times for each different link.

Example Script: post-commit

The example script can be downloaded here. Please remember to remove the .sample extension to enable this example script.

This hook is intended to be used on a private repository and will run after the commit-msg hook has ran successfully. This hook is included as a further example of how to create a Git and RTC work item integration while working in a private repository. Ideally you want to handle work item updates in a public repository by using the post-receive hook. If you add a comment and “Related Artifacts” link in your own private repository, there risks a chance of comments or links being duplicated when pushing it to a public repository. However, if you know the public repository is not set up to update RTC work items, it is possible to do so at this stage. Updating work items at this stage at least ensures that the commit was successful. When coupled with the commit-msg hook, we can also verify that the work item exists, otherwise the commit would abort, and thus this script would not be invoked.

It could be desirable to update a work item even though it currently resides in a private repository. This lets other people in the team who are interested in that work item know in real-time that a new change has been committed, without them having to wait for it to be pushed to a public repository. To correct any duplication issues, we could extend the functionality of the post-receive hook to identify if the exact same “comment” or “link” has already been added to the work item. If it has, then we simply do nothing.

Gitweb

This small section is added to clarify a point that was made earlier. Figure 6 above shows a Gitweb page representing an individual commit in which the comment contains a work item URL that is “underlined” and clickable. As explained previously, this is added because the commit-msg hook appends this string to the comment before the commit is completed. However, by default Gitweb does not convert any links found in a comment into clickable hypertext. To increase the usability of our bidirectional navigation, the “gitweb.cgi” file was modified so that it correctly renders the work item url as a proper HTML link.

This is accomplished in two steps:

First, the subroutine parse_commit_text() is modified so that if a work item URL is found, it will be stored in a variable. We also remove the URL from the Perl @commit_lines array in order to prevent it from being displayed as normal text. This is all done with the following code:

  # Find the Work Item URL  foreach my $line (@commit_lines) {     $line =~ m/^Work Item URL: (https://.*)/;     $work_Item_Url = $1;  }  # Remove the text about the work item URL, we are later going to add a link element	  if ($work_Item_Url) {      # This is safe because commit-msg hook added the URL to the end of the comment     pop (@commit_lines);      pop (@commit_lines);  }

Now that we have the URL stored in a variable, we can simply add the HTML element that will create the link object. In the subroutine git_commit() we simply be added the link element after the comment text is displayed.

  git_print_log($co{'comment'});  if ($work_Item_Url) { #If a work item URL exists, add a link after the comments     print "Work Item URL: ";     print $cgi->a({-href=>$work_Item_Url}, escapeHTML($work_Item_Url));  }

Summary

In this article we have presented three ways to integrate third party SCM systems with RTC. The simplest form of integration is using the Eclipse-based client of your SCM system in the RTC rich client. Another method of integration is using RTC to for your in-house development while contributing patches back to any open source projects that you are making use of. The most in-depth level of integration we discussed is providing links between another SCM system, such as Git, and RTC work items. The example presented was specific to Git but should be easily tailorable to any SCM system that provides server side script based hooks.

Acronyms

  • API: Application Programming Interface
  • CRUD: Create/Read/Update/Delete
  • CVS: Concurrent Versions System
  • HTML: Hypertext Mark-up Language
  • JSON: JavaScript Object Notation
  • MIME: Multipurpose Internet Mail Extensions
  • REST: Representational State Transfer
  • ROA: Resource-Oriented Architecture
  • RTC: Rational Team Concert
  • SCM: Source Control Management
  • SSL: Secure Socket Layer
  • URL: Uniform Resource Locator
  • XML: Extensible Markup Language

References

© Copyright 2009 International Business Machines (IBM)

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

Tue, 23 Jun 2009