Collaborative Lifecycle Management performance report: RDNG 6.0

Introduction

This report compares the performance of IBM Rational DOORS Next Generation version 6.0 to the previous 5.0.2 release. The goal of the test was to verify that the performance of 6.0 operations were the same or better than their 5.0.2 equivalents.

The test methodology involves these steps:

• Run a standard 250 user workload against version 5.0.2 for 1.5 hours (repeat 3 times)
• Run a standard 250 user workload against version 6.0 for 1.5 hours (repeat 3 times)
• Compare the 6 runs.

The 6.0 tests do not include any of the new 6.0 features (such as the configuration management capabilities), which allows the performance workload and data repositories to be the same for 5.0.2 and 6.0.

Findings

The 6.0 release performs the same or better than the 5.0.2 release, when using the 6.0 release with the new configuration management features disabled.

Disclaimer

The information in this document is distributed AS IS. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer’s ability to evaluate and integrate them into the customer’s operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk. Any pointers in this publication to external Web sites are provided for convenience only and do not in any manner serve as an endorsement of these Web sites. Any performance data contained in this document was determined in a controlled environment, and therefore, the results that may be obtained in other operating environments may vary significantly. Users of this document should verify the applicable data for their specific environment.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multi-programming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

This testing was done as a way to compare and characterize the differences in performance between different versions of the product. The results shown here should thus be looked at as a comparison of the contrasting performance between different versions, and not as an absolute benchmark of performance.

Test details

Topology

The topology under test is based on Standard Topology (E1) Enterprise - Distributed / Linux / DB2.

The specifications of machines under test are listed in the table below. Server tuning details listed in Appendix A

Network connectivity

All server machines and test clients are located on the same subnet. The LAN has 1000 Mbps of maximum bandwidth and less than 0.3ms latency in ping.

Data volume and shape

The artifacts were distributed between 6 projects for a total of 517,080 artifacts.

The repository contained the following data:

• 312 modules
• 510,000 module artifacts
• 7,800 requirement artifacts
• 715 folders
• 150 collections
• 0 reviews
• 1,563,492 comments
• 1800 public tags
• 0 private tags
• 1,428 terms
• 1,824,174 links
• 600 views

• JTS index size on disk = 1.2 MB
• RM index size on disk = 41 GB

Each project contained the following data:

• 52 modules
• 119 folders
• 1,180 requirement artifacts
• 85,000 module artifacts
• 14 collections
• 0 reviews
• 260,582 comments
• 300 public tags
• 0 private tags
• 238 terms
• 304,029 links
• 100 views

Methodology

Rational Performance Tester was used to simulate the workload created using the web client. Each user completed a random use case from a set of available use cases. A Rational Performance Tester script is created for each use case. The scripts are organized by pages and each page represents a user action.

Based on real customer use, the test scenario provides a ratio of 70% reads and 30% writes. The users completed use cases at a rate of 60 pages per hour per user. Each performance test runs for 90 minutes after all of the users are activated in the system.

Test cases and workload characterization

Test case execution during test

Response time comparison

The response time charts that follow use the median value (not the average). The median is more consistent than the average in these tests, which makes it easier to compare response times between releases. Response times are reported in milliseconds, where not otherwise specified.

In the repository that contained 517,080 artifacts with 250 concurrent users, no obvious regression was shown when comparing response times between runs. A number of use cases have improved in 6.0.

Results

Observation

• Memory and CPU utilization is similar in 6.0 as in 5.0.2, although there is slightly less CPU utilization on the RM server in 6.0.

Network traffic

• The following chart shows the amount of data sent over the network

Garbage collection

Verbose garbage collection is enabled to create the GC logs. The GC logs show very little variation between runs. Below is an example of the output from the GC logs for each of the JTS and RDNG application servers, including versions 6.0 and 5.0.2.

RDNG

• Observation: 6.0 uses roughly 2G of additional Java heap space. Additional data is cached in memory as part of the 6.0 support for configuration management.
• RDNG 6.0

• RDNG 5.0.2

JTS

• Observation: The graphs show similar behavior in 6.0 as in 5.0.2. The activity is so small that there is not much to graph.

• JTS 6.0

• JTS 5.0.2

Create a collection

Back to Test Cases & workload characterization

Search an artifact using ID and a string

Back to Test Cases & workload characterization

Create, move and delete folders

Back to Test Cases & workload characterization

Create a comment for a module artifact

Back to Test Cases & workload characterization

Open the project dashboard

Back to Test Cases & workload characterization

Create a requirement in a medium sized module

Back to Test Cases & workload characterization

Create a requirement in a large module

Back to Test Cases & workload characterization

Copy and paste 24 artifacts

Back to Test Cases & workload characterization

Create an artifact

Back to Test Cases & workload characterization

Display hover information for an artifact in a collection

Back to Test Cases & workload characterization

Display hover information for a module artifact. Open and edit artifact

Back to Test Cases & workload characterization

Page down in a large module

Back to Test Cases & workload characterization

Change display in a module

Back to Test Cases & workload characterization

Uploading 4 MB file in new artifact

Back to Test Cases & workload characterization

Appendix A

-XX:MaxDirectMemorySize=1g -Xgcpolicy:gencon -Xmx16g -Xms16g -Xmn4g -Xcompressedrefs -Xgc:preferredHeapBase=0x100000000 -Xverbosegclog:logs/gc.log

For more information

• Performance Datasheets: CLM 5.x

About the authors

GustafSvensson

Questions and comments:

• What other performance information would you like to see here?
• Do you have performance scenarios to share?
• Do you have scenarios that are not addressed in documentation?
• Where are you having problems in performance?

Warning: Can't find topic Deployment.PerformanceDatasheetReaderComments