ELM 7.3.0 TRS Resumable Validation Performance Report

Authors: JayrajsinhGohil, SameerkumarKalaria, RamnarayanKumar, JayAkbari, VaughnRokosz
Build basis: IBM Engineering Lifecycle Management (ELM) 7.3.0

Introduction

Resumable Tracked Resource Set (TRS) Validation in IBM Engineering Lifecycle Management (ELM) 7.3.0 enables a validation operation to be paused and resumed without restarting the entire validation process.

This report summarizes the performance and functional validation results for the following scenarios:

• LQE-side validation: Validation of DOORS Next, Engineering Test Management (ETM), and Engineering Workflow Management (EWM) data providers from the Lifecycle Query Engine (LQE) perspective.
• Application-side validation: Validation performed within the DOORS Next, ETM, and EWM applications, covering uninterrupted execution, pause and resume, cancellation, and recovery after an application server restart.

Executive Summary

• Resumable TRS Validation successfully supported pause, resume, cancellation, and restart recovery scenarios across DOORS Next, ETM, and EWM.
• All LQE-side validation runs completed with zero missing and zero stale resources.
• Resumable TRS Validation added no measurable performance overhead to application-side validation runs that were not interrupted.
• LQE-side validation showed no measurable performance impact for ETM and EWM. For DOORS Next, validation took approximately 28% longer (1h 10m to 1h 29m) in the tested environment.
• Across all pause/resume scenarios, resuming validation was significantly faster than restarting the validation process from the beginning.
• Checkpoint storage requirements varied by repository size and application, ranging from approximately 200 MB to 2 GB.

Configuration Properties & Parameters

1. LQE-Side Window Validation Configuration Parameters (Configured on LQE Server)

Window Validation is on by default. You can turn it off with an LQE Advanced Property. Its tuning parameters are set separately for each data provider in LQE administration.

Enabling property (LQE Advanced Properties):

Property	Default value	Purpose
Enable resumable validation	On	Enables or disables resumable validation.

Tuning parameters (LQE administration):

Parameter	Default value	Purpose
Window Count	50,000	Number of resources processed in memory before results are written to disk.
LQE Validation Batch Size	50,000	Number of records read from the LQE database per query.
Merge Fan-In	64	Maximum number of temporary files combined in one step.

2. Application-Side Resumable TRS Validation Properties (Configured on JTS / Application Server)

These properties apply specifically to application-side TRS validation (DOORS Next, ETM, EWM) and are configured through Application-side Advanced Properties:

Property	Purpose
Enable Resumable TRS Validation	Enables or disables resumable validation
Resumable TRS Validation State Directory	Specifies the writable persistent directory used to store validation checkpoint state

Note: For LQE-side validation, configure the settings on the LQE server. For application-side validation, configure the relevant properties in Application-side Advanced Properties.

Standard Disclaimer

The information in this document is distributed AS IS. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer’s ability to evaluate and integrate them into the customer’s operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk. Any pointers in this publication to external Web sites are provided for convenience only and do not in any manner serve as an endorsement of these Web sites. Any performance data contained in this document was determined in a controlled environment, and therefore, the results that may be obtained in other operating environments may vary significantly. Users of this document should verify the applicable data for their specific environment.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multi-programming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

This testing was done as a way to compare and characterize the differences in performance between the existing behaviour and the new behaviour introduced in ELM 7.3.0 for Resumable TRS Validation. The results shown here should be evaluated as comparative performance guidance rather than as an absolute benchmark or SLA for all customer environments.

Key Findings

1. LQE-Side Validation

• Performance impact : Enabling Window Validation had little to no measurable effect on uninterrupted validation for ETM and EWM, with differences within normal run-to-run variability. For DOORS Next, uninterrupted validation took longer by more than that variability. The cause remains an area for future investigation.
  • ETM: 1h 11m 01s with the feature off, 1h 12m 42s with it on (1m 41s, +2.4%) .
  • EWM: 1h 01m 35s with the feature off, 1h 05m 56s with it on (4m 21s,+7.1%).
  • DOORS Next: 1h 09m 47s with the feature off, 1h 29m 08s with it on, which is 19m 21s (+27.7%) longer on a repository of 3.7 million resources.

• Pause and resume: Pausing and resuming added no processing time compared with an uninterrupted run with the feature on. Active processing time, excluding the pause, was 1h 11m (ETM),1h 01m (EWM), and 1h 25m (DOORS Next). The uninterrupted feature-on means were 1h 13m, 1h 06m, and 1h 29m.

• Correctness: All runs ended with zero missing and zero stale resources.

• Storage: The validation directory used up to about 2 GB for about 3.7 million resources.

2. Application-Side Validation

• Performance impact : Enabling the feature had no measurable effect on uninterrupted DOORS Next, ETM and EWM validation. The differences were within normal run-to-run variability.
  • DOORS Next: 56m 15s with the feature off, 51m 26s with it on.
  • EWM: 2h 03m 54s with the feature off, 1h 53m 58s with it on.
  • ETM: 9h 42m to 10h 52m with the feature off, 8h 47m to 10h 42m with it on.

• Pause, resume and restart: DOORS Next and EWM validations resumed from the persisted checkpoint after a pause and after an application server restart. They needed about 11 to 14 minutes of additional processing compared with an uninterrupted run with the feature off. That avoided running the whole validation again, which took about 56 minutes (DOORS Next) or 2 hours (EWM).
  • DOORS Next: 56m 15s uninterrupted with the feature off. After pause and resume, 1h 07m 47s (+11m 32s). After pause, restart and resume, 1h 10m 03s (+13m 48s).
  • EWM: 2h 03m 54s uninterrupted with the feature off. After pause and resume, 2h 15m 52s (+11m 58s). After pause, restart and resume, 2h 18m (+14m 06s).

• Cancellation: Cancelling a paused validation stopped it cleanly, and a new run completed normally.

• Storage: Storage requirements varied depending on the application and repository size. DOORS Next used approximately 200 MB in its state directory for 1.5 million resources, ETM used approximately 2 GB for 3.5 million resources, and EWM used approximately 1.5 GB for 3.8 million resources.

LQE-Side TRS Resumable Testing

Testing was conducted on ELM 7.3.0 with an Oracle database backend and an LQE relational store. The tested data providers and resource counts were:

Provider	TRS endpoint	Resources
ERM	/rm/trs2	3,740,956
QM	/qm/trs2	3,451,996
CCM	/ccm/oslc/workitems/trs2	2,185,699

LQE-side Results summary

The following table summarizes the reported mean validation durations with window validation disabled and enabled.

Provider	Feature OFF	Feature ON	Difference	Finding
ERM	1h 09m 47s	1h 29m 08s	+19m 21s (+27.7%)	Increased validation duration observed
QM	1h 11m 01s	1h 12m 42s	+1m 41s (+2.4%)	Within observed run-to-run variability
CCM	1h 01m 35s	1h 05m 56s	+4m 21s (+7.1%)	Within observed run-to-run variability

ETM and EWM finding: The observed differences were within the reported run-to-run variability. No measurable performance impact was established for these tested scenarios.

DOORS Next finding: Window Validation increased the mean validation duration from 1h 09m 47s to 1h 29m 08s, an observed increase of 19m 21s (+27.7%). The increase exceeded the measured run-to-run variability in the current environment. Further optimization of DOORS Next validation processing remains an area for future investigation.

LQE-side Pause-and-Resume Results

Provider	Total duration, including pause	Pause duration	Active processing time, excluding pause
ERM	1h 39m 30s	Approximately 14m 24s	Approximately 1h 25m 06s
QM	1h 20m 59s	Approximately 10m 18s	Approximately 1h 10m 41s
CCM	1h 15m 08s	Approximately 14m	Approximately 1h 01m 08s

After excluding the planned pause interval, the active processing times were consistent with the corresponding window-validation runs for ETM and EWM. For DOORS Next, the active processing time fell within the observed feature-enabled run range. No additional processing overhead attributable specifically to the pause/resume operation was observed in these tests.

Resource Utilization (LQE-side)

CPU and disk use were low for all three providers. CPU averaged 5–12% on the database server, about 1–1.5% on LQE and 0.5–3% on the application servers. Disk use was negligible except on the database server (7–13%). LQE memory use stayed well below the 32 GB heap.

DOORS Next Application-Side TRS Resumable Testing

Application-side testing used a DOORS Next repository containing approximately 1.5 million resources. Runs used the default setting without clear cache. In interruption scenarios, the reported active validation time excludes the intentional 10-minute pause.

DOORS Next Results Summary

Run	Scenario	Feature state	Active validation time
Baseline	Baseline before feature delivery	Not delivered	53m 23s
1	Uninterrupted validation	OFF	56m 15s
2	Uninterrupted validation	ON	51m 26s
3	Pause, resume, complete	ON	1h 07m 47s
4	Pause, cancel, start fresh	ON	48m 25s
5	Pause, restart server, resume	ON	1h 10m 03s

• No Measurable Performance Overhead: Uninterrupted validation completed in 51m 26s with resumability enabled, compared with 56m 15s with the feature disabled. The results indicate no measurable performance overhead during uninterrupted execution, within the observed run-to-run variability of approximately 3–5 minutes.

• Pause and Resume: Validation successfully resumed from the saved checkpoint after a 10-minute pause and completed in 1h 07m 47s of active processing time, excluding the planned pause interval.

• Server restart recovery: Validation resumed from the persisted checkpoint after the DOORS Next application server restarted. It completed in 1h 10m 03s of active processing time, excluding the planned pause and restart window, which is about 14 minutes more than an uninterrupted run (56m 15s). Rerunning from the start would have repeated the full 56 minutes.

• Cancellation and Fresh Validation: Cancelling a paused validation stopped processing and cleared the existing validation state. A subsequent fresh validation completed successfully in 48m 25s.

• Storage: The application-side validation state directory used approximately 200 MB for the tested 1.5 million-resource repository. A 500 MB to 1 GB allocation was recommended for this test configuration.

ETM Application-Side TRS Resumable Testing

Application-side testing used an ETM repository containing approximately 3.56 million resources. Runs used the default setting without clear cache. For interruption scenarios, active validation time excludes any planned pause intervals.

ETM Results Summary

Run	Scenario	Feature state	Active validation time
Run 1	Uninterrupted validation	OFF	9h 42m 08s
Run 2	Uninterrupted validation	OFF	10h 52m 09s
Run 3	Uninterrupted validation	ON	8h 47m 17s
Run 4	Uninterrupted validation	ON	10h 42m 01s

• No Measurable Performance Overhead: On the large ETM repository of ~3.56 million records, feature-enabled validation completed in 8h 47m–10h 42m, compared with 9h 42m–10h 52m with the feature disabled. Given the natural run-to-run variation of approximately 1–2 hours, the results show no clear measurable performance regression from enabling resumable validation.

• Storage: The application-side validation state directory used approximately 2 GB for the tested 3.5 million-resource repository. A 2-3 GB allocation was recommended for this test configuration.

EWM Application-Side TRS Resumable Testing

Application-side testing used an EWM repository containing approximately 3.8 million resources. Runs used the default setting without clear cache. In interruption scenarios, the reported active validation time excludes the intentional 15-minute pause.

EWM TAP Results Summary

Run	Scenario	Feature state	Active validation time
1	Uninterrupted validation	OFF	2h 03m 54s
2	Uninterrupted validation	ON	1h 53m 58s
3	Pause, resume, complete	ON	2h 15m 52s
4	Pause, cancel, start fresh	ON	1h 27m 23s
5	Pause, restart server, resume	ON	2h 18m

• No Measurable Performance Regression: Uninterrupted validation completed in 1h 53m 58s with resumability enabled, compared with 2h 03m 54s with the feature disabled. The results indicate no measurable performance overhead during uninterrupted execution, within the observed run-to-run variability of around 30 minutes.

• Pause and Resume: Validation successfully resumed from the saved checkpoint after a 15-minute pause and completed in 2h 15m 52s of active processing time, excluding the planned pause interval.

• Server restart recovery: Validation resumed from the persisted checkpoint after the EWM application server restarted. It completed in 2h 18m of active processing time, excluding the planned pause and restart window, which is about 14 minutes more than an uninterrupted run (2h 03m 54s). Rerunning from the start would have repeated the full 2 hours.

• Cancellation and Fresh Validation: Cancelling a paused validation stopped the existing run. A subsequent fresh validation completed successfully in 1h 27m 23s.

• Storage: The application-side validation state directory used approximately 1.5 GB for the tested 3.8 million-resource repository. A 2-3 GB allocation was recommended for this test configuration.

Test Environment Details & Hardware Specifications

The performance test topology comprises physical enterprise servers hosting the complete ELM 7.3.0 solution stack:

Test Environment for LQE- and ETM-Side Resumability Testing

Server Role	Operating System	Application	Virtual Cores	RAM
Proxy Server	RHEL Server 7	IHS Server	8 Total	16 GB
JTS / GC	RHEL Server 8	JTS/GC Server	24 Total	32 GB
ELM Application Server 1	RHEL Server 7	ERM50/QM50/CCM50	24 Total	32 GB
ELM Application Server 2	RHEL Server 7	ERM51/QM51/CCM51	24 Total	32 GB
LQE rs Server	RHEL Server 9	LQE Server	24 Total	128 GB
Database Server	RHEL Server 8	Oracle DB	24 Total	64 GB

Test Environment for DOORS Next- and EWM-Side Resumability Testing

Server Role	Operating System	Application	Virtual Cores	RAM
Web / Reverse Proxy	RHEL 7.9	IHS for LQE Environment	16 Total	16 GB
DOORS Next Server	RHEL 7.9	DOORS Next Server	24 Total	64 GB
CCM Server	RHEL 7.9	CCM	24 Total	32 GB
JTS Server	RHEL 7.9	JTS	24 Total	96 GB
DB2 Database Server	RHEL 7.9	DB2 Database	24 Total	64 GB
ETM/QM Server	RHEL 7.9	ETM/QM	24 Total	64 GB
Report Builder and DB2 Server	RHEL 7.9	Report Builder + DB2	24 Total	64 GB
LQE Server	RHEL 7.9	LQE	40 Total	768 GB
ELM Application Server 1	RHEL 7.9	QM3/CCM3/DOORS Next3	24 Total	32 GB
ELM Application Server 2	RHEL 7.9	QM2/CCM2/DOORS Next2	24 Total	32 GB
Report Builder Server	RHEL 7.9	Report Builder	24 Total	32 GB
Oracle and LQE Server	RHEL 7.9	LQE + Oracle Database	40 Total	768 GB
LDX Server	RHEL 8.x	LDX Server	24 Total	64 GB
Oracle/LDXDB/LQE rs Database Server	RHEL 8.x	Oracle DB/LDXDB/LQE rs	40 Total	768 GB

Abbreviations

• JTS: Jazz Team Server
• GCM: Global Configuration Management
• ERM: Engineering Requirements Management
• ETM / QM: Engineering Test Management / Quality Management
• EWM / CCM: Engineering Workflow Management / Change and Configuration Management
• TAP: Tracking and Planning
• LQE rs: Lifecycle Query Engine (relational store)
• TRS: Tracked Resource Set
• SLA: Service Level Agreement