Performance report for Rational Quality Manager 6.0: configuration management

Introduction

The 6.0 release of IBM® Rational® Quality Manager (RQM) introduces a major new feature: configuration management. You can now manage the evolution of test artifacts using techniques similar to those which Rational Team Concert supports for managing source code. You can organize test artifacts into streams and baselines. You can easily reuse test artifacts by branching streams or baselines, and you can apply versioning across your entire CLM solution by using global configurations.

This 6.0 report focuses on the new configuration management feature. It examines how response times are impacted by when adopting configuration management. It also looks at the how response times are impacted by the number of streams or baselines.

If you are not using configuration management, please refer to Collaborative Lifecycle management report: Rational Quality Manager 6.0 release. For more information on global configurations, please refer to Collaborative Lifecycle Management performance report: Global configuration and back link indexing.

Standard disclaimer

The information in this document is distributed AS IS. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk. Any pointers in this publication to external Web sites are provided for convenience only and do not in any manner serve as an endorsement of these Web sites. Any performance data contained in this document was determined in a controlled environment, and therefore, the results that may be obtained in other operating environments may vary significantly. Users of this document should verify the applicable data for their specific environment.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the number of other applications running in the customer environment, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

This testing was done as a way to compare and characterize the differences in performance between different configurations of the 6.0 product. The results shown here should thus be looked at as a comparison of the contrasting performance between different configurations, and not as an absolute benchmark of performance.

Test methodology

IBM® Rational® Performance Tester (RPT) was used to simulate traffic generated from a browser client. RPT test scripts were created for each use case. The scripts are organized by pages where each page represents a user action; Tests were run with a 60 second “think time” between pages for each user.

The test environment is described in section Topology and Key configuration parameters. Test repositories were migrated from 5.0.2GA, but then the RQM projects were enabled for configuration management and additional streams and baselines were created.

• The layout for sub-streams and baselines in the initial stream migrated is shown in the picture below

Additional details on the specific data shapes tested can be found in the Data Volume and shape section of Appendix A.

Impact of working in streams and baselines

We conducted a series of tests to assess the impact of using the new configuration management features. We looked at common RQM "read" use cases under three different conditions:

• With configuration management disabled
• With configuration management enabled, in the context of a stream
• With configuration management enabled, in the context of a baseline. Here we looked only at read operations, since a baseline does not allow changes.

The charts below show the detailed results for a 1000 user workload in repository with 500K test artifacts. Here is a summary:

• There is a small cost to enabling configuration management. Most operations slow down slightly (< 250 ms) after enabling configuration management.
• There is an additional cost when running in the context of a baseline (as opposed to a stream) for several operations (artifact browsing, searching, and sorting). This operations can be as much as 250 ms slower in a baseline than in a stream, and as much as 500 ms slower in a baseline than with configuration management disabled.

Impact of artifact counts

We next looked at how response times were impacted by increasing the number of artifacts. We looked at two things:

• the number of streams in a project area
• the total number of artifacts in a project area

Number of streams

In this test, we created additional streams from the initial stream, to look at how response times were impacted as the number of streams increased. The initial stream contained 500,000 test artifacts (as did all of the additional streams). What we found was that:

• Create or edit operations were not impacted by the number of streams.
• Some read operations (browsing test execution records and test scripts) became slower as the number of strames increased.

Selected page response times for reads as a function of number of streams

Memory and CPU usage on the database server increased as the number of streams increased.

Resource utilization as a function of number of configurations

CPU	Memory

Total number of artifacts

What was found was that:

• Response times degrade as the number of artifacts in a project area increases.
• Larger repositories use more memory, CPU, and have higher disk I/O than smaller repositories
• The response times in the 4M repository have degraded to a point where that system would be difficult to use.

Average page response time for all pages as function of size of the project area

Response time charts for selected pages as function of size of the project area

page response time comparisons

It was also observed that the time to create the first substream increased significantly as the size of the project area grew.

Substream creation time as function of size of the project area

System resource utilization is increased as the artifact counts are getting larger -

OS util as function of size of the project area

CPU	Disk	Memory

Appendix A:

Topology

The topology under test is based on Standard Topology (E1) Enterprise - Distributed / Linux / DB2.

The specifications of machines under test are listed in the table below. Server tuning details listed in Appendix B

N/A: Not applicable. vCores = Cores with hyperthreading

Network connectivity

All server machines and test clients are located on the same subnet. The LAN has 1000 Mbps of maximum bandwidth and less than 0.3 ms latency in ping.

Data volume and shape

The artifacts were migrated from 5.0GA repository which contains a total of 579,142 artifacts in one large project.

The repository contained the following data:

• 60 test plans
• 30,000 test scripts
• 30,000 test cases
• 120,000 test case execution records
• 360,000 test case results
• 3,000 test suites
• 5,000 work items(defects)
• 200 test environments
• 600 test phases
• 30 build definitions
• 6,262 execution sequences
• 3,000 test suite execution records
• 15,000 test suite execution results
• 6,000 build records

• QM Database size = 12 GB
• QM index size = 1.5 GB

User roles, test cases and workload characterization

Test Cases

Appendix B: Key configuration parameters

-Xgcpolicy:gencon -Xmx8g -Xms8g -Xmn2g  -Xss786K
-Xcompressedrefs -Xgc:preferredHeapBase=0x100000000 -verbose:gc -XX:+UseParallelGC 
-Xverbosegclog:gc.log -XX:MaxDirectMemorySize=1G

Appendix C: Enabling configuration management (CM) for QM

Please refer to the article Collaborative Lifecycle Management performance report: Global configuration and back link indexing for information on what to expect when first activating the configuration management feature, and for information the performance of linking scenarios test artifacts and other tools like IBM Rational DOORS Next Generation or IBM Rational Team Concert.

Appendix D: System resource usage - with and without configuration management enabled

Resource utilization

Resource utilization for 6.0 - with vs without CM

Garbage collection - JVM GC Chart

Verbose garbage collection is enabled to log the GC activities. These GC logs showed very little variation between runs. There is also no discernible difference between versions. Below is one example of the output from the GC log for each application.

QM WAS server - with vs without CM

With CM enabled	Without CM enabled

JTS WAS server - with vs without CM

With CM enabled	Without CM enabled

For more information

• Collaborative Lifecycle Management performance report: Rational Quality Manager 6.0 release
• Collaborative Lifecycle Management performance report: Rational Quality Manager 5.0 release
• Collaborative Lifecycle Management performance report: Global configuration and back link indexing
• Rational solution for Collaborative Lifecycle Management V6.0 documentation in IBM Knowledge Center

About the authors:

Related topics:

Questions and comments:

• What other performance information would you like to see here?
• Do you have performance scenarios to share?
• Do you have scenarios that are not addressed in documentation?
• Where are you having problems in performance?

Warning: Can't find topic Deployment.PerformanceDatasheetReaderComments