ELM 7.3.0 TRS Validation Performance Report

Introduction

In the IBM Engineering Lifecycle Management (ELM) 7.3.0 release, significant performance improvements were made to Tracked Resource Set (TRS) validation across multiple ELM applications — Requirements Management (RM), Engineering Test Management (ETM), and Engineering Workflow Management (EWM). These improvements reduce the time required to complete TRS validation, thereby reducing the operational overhead for customers who run TRS validation as part of their maintenance or troubleshooting workflows.

This article presents the performance test results for TRS validation improvements in ELM 7.3.0 for the following applications:

• DOORS Next
• Engineering Test Management (ETM)
• Engineering Workflow Management (EWM)

Standard Disclaimer

The information in this document is distributed AS IS. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk. Any pointers in this publication to external Web sites are provided for convenience only and do not in any manner serve as an endorsement of these Web sites. Any performance data contained in this document was determined in a controlled environment, and therefore, the results that may be obtained in other operating environments may vary significantly. Users of this document should verify the applicable data for their specific environment.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multi-programming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

This testing was done as a way to compare and characterize the differences in performance between different versions of the product. The results shown here should thus be looked at as a comparison of the contrasting performance between different versions, and not as an absolute benchmark of performance.

Key Findings

When running TRS validation from the application side, there are two modes: with and without the "clear cache" option. Without clear cache, the application uses its previously cached TRS state as a starting point and only validates changes since the last run — this is the recommended mode for routine use and completes much faster. With clear cache, the application discards its cached TRS state and re-validates the entire TRS feed from scratch, processing every resource in the repository regardless of whether it has changed. This is a more thorough but significantly more time-consuming operation, typically used after upgrades or database restores where the cached state can no longer be trusted.

The following summarizes the key performance findings from TRS validation testing in ELM 7.3.0:

• DOORS Next: TRS validation time for a DOORS Next repository of approximately 1.5 million records and 1,126 baselines was reduced from 8 hours 9 minutes 44 seconds to 53 minutes 23 seconds. This represents approximately a 9x improvement. Testing was performed in the recommended mode (without the "clear cache" option), so this improvement applies to normal routine validation.

• ETM: Enabling the Skip Step Result from TRS Processing property reduced TRS resource volume by nearly 50% (from approximately 6.7 million to 3.45 million resources). TRS validation time was reduced from 2 days 2 hours 49 minutes to 1 day 7 hours 17 minutes (~38% faster). Testing was performed without the "clear cache" option, so this improvement applies to normal routine validation.

• EWM: TRS validation time for an EWM repository of approximately 2.2 million records was reduced from 3 hours 51 minutes 26 seconds to 1 hour 28 minutes 22 seconds, representing approximately a 62% improvement. However, this improvement was measured using the "clear cache" option (full re-processing of the TRS feed). No significant improvement was observed when validating without the clear cache option (i.e., using the cache). Customers should not expect TRS validation performance gains for EWM in routine cached validation mode.

Upgrading to ELM 7.3.0 can significantly reduce TRS validation time, depending on the application and validation mode. The largest gains were observed for DOORS Next routine validation and EWM clear-cache validation, while ETM improvements depend on enabling the Skip Step Result from TRS Processing property.

Performance Summary

DOORS Next TRS Validation Performance

Performance Test Results

Summary of Results

• TRS validation time was reduced from approximately 8 hours 10 minutes to approximately 53 minutes — a reduction of over 7 hours.
• This is approximately a 9x improvement in validation speed for a repository of ~1.5 million records and 1,126 baselines.
• The improvement is available by default in ELM 7.3.0. No additional configuration or enabling of advanced properties is required.
• Testing was performed with Oracle as the database backend, without the clear cache option.

Test Methodology

This section describes the data used for testing, the test environment, and the approach taken to measure TRS validation performance.

Data Shape

TRS validation performance testing for DOORS Next was performed against a repository with the following characteristics:

Testing Approach

TRS validation was triggered manually against the DOORS Next application. The test compared the time taken to complete a full TRS validation run before and after the 7.3.0 performance improvement, using the same repository and the same server environment.

Key testing parameters:

• Database: Oracle
• TRS validation mode: Standard validation (without the "clear cache" option)
• Comparison: Pre-improvement baseline (representing behavior prior to the 7.3.0 fix) vs. post-improvement (7.3.0)

Test Environment Details

ETM TRS Validation Performance

Performance Test Results

Summary of Results

With Skip Step Results enabled, TRS resource volume was reduced from approximately 6.7 million to 3.45 million resources — a reduction of nearly 50%. This directly translated into the following performance improvements:

• ETM TRS rebase: Reduced from approximately 1 day 15 hours to approximately 24 hours, a reduction of about 15 hours (38.8% faster).
• ETM TRS validation: Reduced from approximately 2 days 3 hours to approximately 1 day 7 hours, a reduction of about 19.5 hours (38.4% faster).

The Skip Step Result from TRS Processing property controls whether ETM step results are included in the TRS feed. Enabling this option reduces the volume of resources in the TRS feed, which directly reduces the time required for TRS rebase and TRS validation operations. The degree of improvement will depend on the proportion of step results relative to other artifacts in your repository — deployments with a high ratio of step results will see the greatest benefit. For full details on the tradeoffs and how to enable this setting, see the IBM documentation: Skip Step Result from TRS Processing.

Unlike the DOORS Next and EWM results, the ETM improvement comes from reducing TRS resource volume by excluding step results from TRS processing. The performance gain is therefore dependent on repository content and configuration.

Detailed Results

Test Methodology

Testing in the Performance Verification Test (PVT) environment compared the same operations before and after enabling ETM Skip Step Results. Both runs used the same ETM repository, test scenarios and execution sequence.

The following TRS operations were measured:

• ETM TRS rebase
• ETM TRS validation

Data Shape

The same ETM repository was used for both runs. The repository contained about 243 million versions, including about 5.2 million step-result version states and about 7.6 million version states for everything else.

Testing Approach

The same test sequence was followed for both Skip Step Results Disabled and Enabled runs:

• Capture the initial TRS Console metrics and repository statistics.
• Perform an ETM TRS rebase and record the completion time.
• Perform ETM TRS validation and record the completion time.
• Capture the final TRS Console metrics and repository statistics for comparison.

The baseline run was performed with the Skip Step Result from TRS Processing property set to its default value of false. For the comparison run, the property was set to true before performing the ETM TRS rebase.

Test Environment Details

EWM TRS Validation Performance

Performance Test Results

Summary of Results

• EWM TRS validation time was reduced from 3 hours 51 minutes 26 seconds to 1 hour 28 minutes 22 seconds when running validation with the "clear cache" option enabled — a reduction of over 2 hours 23 minutes (~62% faster).
• The "clear cache" option forces a full re-processing of the entire TRS feed, discarding any previously cached index state. This is typically used after upgrades or database restores.
• No significant improvement was observed when running EWM TRS validation without the clear cache option (i.e., in the normal cached mode). Customers running routine validation should not expect a performance gain for EWM in ELM 7.3.0.
• The improvement is available by default in ELM 7.3.0. No additional configuration or enabling of advanced properties is required.
• Testing was performed with Oracle as the database backend.

Test Methodology

This section describes the data used for testing, the test environment, and the approach taken to measure TRS validation performance of EWM.

Data Shape

TRS validation performance testing for EWM was performed against a repository with the following characteristics:

Testing Approach

TRS validation was triggered manually against the EWM application. The test compared the time taken to complete a full TRS validation run before and after the 7.3.0 performance improvement, using the same repository and the same server environment.

Key testing parameters:

• Database: Oracle
• TRS validation mode: Validation with the "clear cache" option enabled. This option clears the existing LQE index and re-processes the entire TRS feed from scratch, rather than using the cached index state as a starting point.
• Comparison: Pre-improvement baseline (representing behavior prior to the 7.3.0 fix) vs. post-improvement (7.3.0)

Test Environment Details

Abbreviations

• CCM = Change and Configuration Management (Engineering Workflow Management SCM component)
• ELM = Engineering Lifecycle Management
• ETM = Engineering Test Management
• EWM = Engineering Workflow Management
• GCM = Global Configuration Management
• IHS = IBM HTTP Server
• JTS = Jazz Team Server
• LDX = Link Index Provider
• LQE = Lifecycle Query Engine
• QM = Quality Management (ETM application context)
• RM = Requirements Management (DOORS Next application context)
• TRS = Tracked Resource Set