How Rational SVT uses in-house deployments to test Collaborative Lifecyle Management products

Authors: Joe Quigley - (jquigley@us.ibm.com)
Build basis: Rational Team Concert, Rational Quality Manager, Rational Requirements Composer and RRDI/Rational Insight

Challenge

It has always been IBM Rational's goal to test pre-release versions of Rational products in a production environment before the product ships to customers. Rational's System Verification Team (SVT) looked at the challenge of how to test pre-release builds of CLM, with an emphasis on Rational Quality Manager (RQM), first in a production-like test bed, and then in production itself. The staging environment needed to provide:

1. Real production data
2. Integrations among RQM, Rational Team Concert (RTC), Rational Requirements Composer (RRC), and legacy applications
3. Compatibility between different CLM versions
4. Enterprise reporting with CLM and Rational Reporting for Development Intelligence//Rational Insight using a common data warehouse.

This staging environment also needed to offer a robust test scenario that provided confidence that the early builds could be safely deployed to production. The following sections describe how this challenge was met.

The SVT in-house deployment staging environment

A primary goal for In-House Deployment (IHD) staging is to make it look as much like production as possible. By using the same data, projects, and connections, there is a high degree of probability that any product defects can be found before they hit production, or more importantly, customers.

Setting up the application servers

IHD staging consists of 3 Jazz Team Servers, 4 Rational Team Concert servers, a Rational Quality Manager server, an Rational Requirements Composer server, a Rational Reporting for Development Intelligence/Insight Extract Transform and Load (ETL) server, an Rational Reporting for Development Intelligence/Insight report server, and a ClearQuest server. These are hosted on 6 virtual machines running on VMware ESX servers. Each of these products being further segregated into application instances with separate IP addresses.

• JTS/RTC/RQM/RRC servers - Split between two Linux RedHat Server 5 systems. A unique IP address is assigned to these application instances. The systems are virtual machines running with 2 CPUs, 8GB memory, and 150GB of free disk space.
• RRDI/Insight servers - Two Windows 2003 servers hosting RRDI/Insight. One system hosts the Insight ETL server; the other hosts the report server. Each system is a virtual machine running Windows 2003, with 2 CPUs, 8GB memory, and 170GB of free disk space.
• ClearQuest (CQ) server - Windows 2003 system hosting CQ. This system hosts a CQ replica, including its own database running MS-SQL. The system is a virtual machine running Windows 2003, with 2 CPUs, 8 GB of memory, and 170GB free disk space for the application and 100GB free space for the CQ database.

In order to give the appearance of the real production environment, a common host file is added to all the staging servers to map production server names to the staging IP addresses. For example:

Test IP Production Server 9.77.777.7 jts01.ibmrational.com 9.99.999.9 jazzrqm1.ibmrational.com 9.88.888.8 jazzrtc1.ibmrational.com

The same host file is supplied to all testers so that any apparent production application urls will actually point to the staging application servers. For example, production url https://jazzrqm1.ibmrational.com/jazz/web/console/productA takes the tester to a duplicate of that production project, but in staging.

Setting up the data

To provide production data quantity and quality, production backups of the application repositories and the common data warehouse are restored to staging. These large DB2 databases are set up on two dedicated DB servers, one AIX, the other Linux. Since the application servers are equipped with the necessary host file, there is no additional work required for the data.

• stagedb1 - Linux RedHat Server 5 hosting DB2. The system is a virtual machine with 2 CPUs, 6GB memory, and 420GB drive, of which 300GB has been partitioned for database storage.
• stagedb2 - AIX 6.1 system hosting DB2. The system is a P-Series LPAR with 2/4 cores, 8GB memory, and 800GB for database storage, and 1.6TB for backup database storage.

Note: Using the CLM Server Rename feature would have been another viable option for building the SVT staging environment.

Working with multiple CLM versions

In large enterprises, products are typically upgraded in a staged manner. Staging upgrades are performed so that the latest pre-release software is always matched with one or more already-released versions. For example, if the current pre-release version is CLM 4.0.3, then it is matched with CLM 4.0.2 and CLM 4.0.1 servers to make sure that backward compatibility is successful.

Integrations

In staging, Rational Quality Manager to Rational Team Concert, Rational Team Concert to Rational Team Concert, Rational Quality Manager to Rational Requirements Composer, and Rational Team Concert to ClearQuest integrations are set up and tested to exercise the relationships between CLM applications and between CLM and legacy applications. The bi-directional arrows in the environment diagram represent these relationships.

Enterprise reporting

SVT uses Rational Rational Reporting for Development Intelligence//Insight to report on the testing metrics that help make product decisions. In staging, the latest versions of Rational Reporting for Development Intelligence//Insight and CLM are configured and tested. If necessary, common data warehouse schema upgrades are performed. CLM packages are imported into the Rational Reporting for Development Intelligence/Insight Data Manager, and the CLM reporting xdc files are downloaded and merged. Extract Transform and Load (ETL) processes are run against all CLM applications. On the reporting side, SVT-specific customizations are made to both the FM model and reports. Testing includes:

1. Insight reporting
2. CLM BIRT reporting against the data warehouse
3. CLM reporting against live data in the application repositories.

Test Cycles

During any given CLM product development cycle, SVT tests early milestones, release candidates, and the final GA (customer) release in this IHD staging environment. Each of these milestones and candidates consists of multiple builds that are tested. When the final build of a given milestone or candidate is approved, that build is pushed to production, where it is used by 3000 users on the target servers during daily business operations.

During the test cycles, SVT works very closely with Development to uncover and fix problems. If a defect is found in IHD staging, a workitem is quickly submitted, very often by using an earlier pre-release CLM version that has already been promoted to production. Development analyzes and fixes the defect and pushes the fix to a new build that is then applied to staging. Once a CLM version has been released, the product test cycle starts over with the next planned version. This continuous improvement process continues as Rational uses heavily the product that is eventually shipped to customers.

About the author

Joe Quigley is part of the RES Jazz Operations team which is responsible for two site environments operating several CLM deployments in various configurations and having an IBM base of over 7000 active users. Joe is the RES team lead for SVT in-house deployment of pre-GA CLM product releases. He has a background in software change management, multisite database operations, and product management.

Related topics: Deployment web home, Deployment web home

External links:

• IBM

Additional contributors: TWikiUser, TWikiUser