ETM Read Access Control (RAC) Performance Report

Introduction

Engineering Test Management (ETM) Read Access Control (RAC) provides granular visibility control within a single, non-configuration-management (Non-CM) project area by associating read access permissions with Jazz Team Areas. Team members only see the test artifacts assigned to the team areas in which they hold membership (or child team areas thereof), while unauthorized users are prevented from viewing or querying restricted artifacts.

Note: Read-access control is not supported for project areas that have configuration management (CM) enabled.

Supported Test Artifacts

Read access control using team areas is supported across the core ETM artifact lifecycle:

• Test Plan (TP)
• Test Case (TC)
• Test Suite (TS)
• Test Script
• Test Case Execution Record (TCER)
• Test Suite Execution Record (TSER)
• Test Case Result
• Test Suite Result
• Test Data
• Keyword
• Attachment

Assignment of a team area for the purpose of read access control can be performed on individual test artifacts in their respective editors or in bulk directly from artifact browsing tables.

For additional feature documentation, refer to the IBM Documentation on ETM Read Access Control by Using Team Areas.

Standard Disclaimer

The information in this document is distributed AS IS. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer’s ability to evaluate and integrate them into the customer’s operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk. Any pointers in this publication to external Web sites are provided for convenience only and do not in any manner serve as an endorsement of these Web sites. Any performance data contained in this document was determined in a controlled environment, and therefore, the results that may be obtained in other operating environments may vary significantly. Users of this document should verify the applicable data for their specific environment.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multi-programming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

This testing was done as a way to evaluate and characterize the performance, scalability, and resource utilization of ETM Read Access Control under high concurrent user load and large data volumes. The results shown here should be evaluated as comparative performance data and operational guidance rather than an absolute benchmark for all custom deployments.

Key Findings

Performance and scalability characterization of ETM Read Access Control (RAC) by using Team Areas across large enterprise repositories (147M+ artifacts) established the following key conclusions:

• High Concurrency Scalability:
  • With up to 500 team areas, RAC scales reliably up to 3,000 concurrent users without encountering a performance degradation inflection point.
  • Under standard workloads (up to 1,500 users), all transactions consistently satisfied the 5.0-second Service Level Agreement (SLA) threshold.

• Low Infrastructure Overhead:
  • The RAC SQL access-filtering logic introduces minimal CPU overhead on the hosting environment. At peak 1,500-user load, CPU utilization remained well below capacity—peaking at 12.4% on the Oracle 19c Database and 8.7% on the ETM Liberty Application Server.

• Team Area Sizing Boundaries:
  • Because RAC evaluates team hierarchies dynamically in SQL queries, high team area density per project introduces sizing boundaries:
    • 500 Team Areas: Recommended up to 3,000 concurrent users.
    • 1,000 Team Areas: Recommended up to 1,800 concurrent users (degradation inflection point at ~2,000 users).
    • 2,000 Team Areas: Recommended up to 1,000 concurrent users (elevated latency beyond 1,000 users).

• Identified Bottlenecks:
  • Initial project area dashboard loads invoke /processAreaHierarchiesWithApplicationContext, which can take up to ~1 minute once team area counts exceed approximately 1,000 — as this service computes the full project and team area hierarchy on every dashboard load.
  • User permission resolution (/processAreasForUser) scales linearly with the number of team areas assigned to an individual user, warranting the practice of least privilege in user team assignments.

Test Methodology

This section details how the ETM RAC performance characterization and scale tests were conducted, including the repository data shape, environment topology, test scenarios, and workload execution methodology.

Data Shape & Repository Scale

Performance validation was executed against our largest ETM repository containing approximately 147M artifacts across 2,040 project areas and 20,200 team areas.

To stress-test the new RAC SQL filtering paths, hierarchy resolution logic, synthetic benchmark project areas and team area hierarchies were generated in two structured phases using the Java-based ETM Data Generator tool:

• Phase 1 — Project & Team Area Hierarchy Generation:
  • Created ~1,000 non-configuration ETM project areas.
  • Configured hierarchical team structures in each project: 10 top-level team areas, each having 1 sub-team area (20 team areas per project).
  • Assigned users across project and team areas with standard JazzUsers and JazzAdmin roles.

• Phase 2 — High-Volume RAC Project Areas:
  • Populated high-volume project areas generating ~250K to ~2.3M total artifacts with complex team area assignments.
  • Configured user populations from 500 up to 10,000 assigned users per project area.
  • Generated multiple artifact states and relationship links across Test Plans, Test Cases, Test Suites, Test Scripts, and Execution Records.

The following screenshot illustrates the team area hierarchy configuration established in the test project areas:

Workload & Test Scenarios

The performance workload was designed to simulate realistic end-user navigation patterns, browsing actions, and complex filtering operations across RAC-enabled repositories. The core scenario suite included:

1. Authentication & Dashboard Navigation:
   • RQM_Login: User authentication and initial session handshake.
   • RQM_Dashboard: Initial project dashboard rendering and widget data retrieval.
2. Test Plan & Artifact Browsing:
   • QM_Browse_TestPlans: Loading the paginated Test Plan table.
   • QM_Open_Random_TestPlan: Opening a full Test Plan editor.
   • QM_Open_TP_TC_View: Browsing Test Cases associated with a selected Test Plan.
   • QM_Open_TP_TS_View: Browsing Test Suites associated with a selected Test Plan.
   • QM_Open_TP_TCER_View: Browsing Test Case Execution Records (TCERs) associated with a selected Test Plan.
   • QM_Open_TCs_List_View: Browsing the main Test Cases table view.
   • QM_Open_TCERs_List_View: Browsing the main Test Case Execution Records (TCERs) table view.
3. Direct Artifact Opening:
   • QM_Open_Random_TestCase, QM_Open_Random_TestSuite, QM_Open_Random_TCER.
4. TCER Filter Operations:
   • Iterations
   • Last Result Passed
   • Last Result Failed
   • Blocked / Error
   • Owners
   • Owners + Weights
   • Owners Exclude Iterations
   • Test Environment + Owners
5. Test Case Filter Operations:
   • Categories
   • Categories Excluding
   • Categories (Some Unassigned)
   • Draft Status
   • Created By
   • Owners
   • Categories + Test Plan Script Steps
   • Categories + Test Suites + Test Scripts
   • Categories + Weight

Load Testing Methodology

Multi-user workload simulation was executed using Rational Performance Tester (RPT):

• Multi-user load schedules were built to execute mixed-scenario workloads with realistic pacing and think time.
• Multi-tier scale runs were conducted from 10 concurrent users up to 1,500 concurrent users (with dedicated scaling capacity evaluations up to 3,000 users).
• Workloads were executed side-by-side against both Non-CM (RAC) and CM-Aware project areas under identical hardware and database configurations to isolate and quantify the RAC access-filtering overhead.

The following screenshot shows the RPT test schedule structure:

Monitoring Tools

Comprehensive monitoring was active throughout all benchmark windows:

• NMON (Nigel's Monitor): Continuous recording of CPU, memory utilization, disk I/O, and network throughput across all physical nodes.
• Oracle AWR (Automatic Workload Repository): Profiling database execution time, RAC Common Table Expression (CTE) query overhead, buffer cache hits, and SQL execution plans.
• Java Diagnostics: Thread dumps, javacores, and queryStats instrumentation to detect internal lock contention or JVM memory pressure.
• RPT Report: Aggregating transaction throughput, response times, and error rates.

Test Environment Details

The performance tests were conducted on an enterprise four-server topology deployed on bare-metal physical servers.

Performance Test Results

This section presents the performance test findings, capacity thresholds, comparative metrics between Non-CM (RAC) and CM-Aware project areas, and server resource utilization under load up to 1,500 concurrent users.

Summary of Load Test Results

• Stable Scalability to 1,500 Users: ETM Non-CM project areas with Read Access Control enabled scale reliably up to 1,500 concurrent users with zero response time threshold breaches (> 5.0 seconds SLA).
• Minimal Server Resource Overhead: Server CPU utilization remained exceptionally low on both tiers at 1,500 concurrent users—peaking at 12.4% on the Oracle Database server and 8.7% on the ETM Application server.
• Fast Navigation & TCER Filtering in Non-CM: Browsing Test Plans (~236 ms), opening TCER views (~328 ms), and executing TCER filters (285–307 ms) demonstrated sub-second response times across all load levels.
• Predictable Scaling Profile: Response times for navigation, dashboards, and filtering remained flat from 100U to 1,500U, demonstrating strong query execution stability.

Load Testing with Varying Team Area Counts (Capacity Limits)

To establish operational sizing guidelines, dedicated load testing was conducted with varying numbers of team areas (500, 1,000, and 2,000 team areas) under increasing concurrent user counts. Because RAC SQL filters evaluate team hierarchy paths and memberships during query execution, team area density directly influences database query planning and memory structures.

The testing revealed clear scaling thresholds across the evaluated team area topologies:

1. 500 Team Areas

• Tested Load Range: Scaled up to 3,000 concurrent users.
• Behavior: Response times scaled predictably across the entire load spectrum without encountering a performance degradation inflection point.
• Compliance: All transaction response times consistently satisfied operational SLA criteria.
• Recommended Operational Limit: Up to 3,000 concurrent users.

2. 1,000 Team Areas

• Stable Load Range (100 to 1,600 Users): Performance remained completely stable across all transactions, well within acceptable SLA limits.
• Degradation Threshold (1,600 to 2,000 Users): Starting around 1,800 concurrent users, response times began experiencing exponential growth due to cumulative database CTE resolution overhead and lock queues under high team area density.
• Recommended Operational Limit: Up to 1,800 concurrent users.

Chart 1: Response Time vs Users (100 to 1,600 Users — Stable Performance)

Chart 2: Response Time vs Users (1,600 to 2,000 Users — Degradation from 1,800U)

3. 2,000 Team Areas

• Stable Load Range (100 to 1,000 Users): Response times remained predictable and within acceptable operational thresholds up to 1,000 concurrent users.
• Degradation Threshold (1,000 to 1,400 Users): Exponential performance degradation and elevated query latency were observed immediately past 1,000 users as massive team area volume magnified the cost of hierarchy evaluation under concurrent load.
• Recommended Operational Limit: Up to 1,000 concurrent users.

Chart 1: Response Time vs Users (100 to 1,000 Users — Stable Performance)

Chart 2: Response Time vs Users (1,000 to 1,400 Users — Degradation beyond 1,000U)

4. Summary of Sizing Guidelines by Team Area Scale

Server CPU Utilization

Resource monitoring confirmed that neither the ETM application server nor the Oracle database server experienced CPU exhaustion during the 1,500 user scale test. Both Non-CM (RAC) and CM-Aware workloads exhibited similar, moderate CPU growth profiles.

The charts below show the CPU utilization curves on the ETM Application Server and Database Server as user load scaled from 10 to 1,500 concurrent users:

Detailed Response Time Analysis: Non-CM (RAC) vs CM-Aware

The multi-user benchmark measured 31 distinct user transactions and queries across both project area modes.

1. Navigation & View Operations

2. TCER Filter Operations

TCER filtering demonstrated high efficiency under RAC in Non-CM project areas, consistently executing in 280 ms to 380 ms. In CM-Aware project areas, TCER queries executed in the 1.7 s to 2.0 s range due to configuration-context resolution overhead across versioned configurations.

3. Test Case Filter Operations

In Test Case filtering, queries involving negative category filters (Categories Excluding) or unassigned status (Draft) in Non-CM RAC project areas require broader table scans and multi-join CTE evaluations across the large unversioned artifact repository. These operations took ~4.2–4.3 s at low load (cold cache) and stabilized at ~3.0–3.1 s at 1,500 users. In CM-Aware project areas, the versioned component scoping allows these queries to execute faster (~150 ms).

Performance Limitations

During rigorous performance and single-user scale profiling, two existing Jazz Foundation (JAF) Web UI services were identified as performance bottlenecks when the total number of project areas, team areas, and user memberships in an environment becomes very large.

Important Technical Context: It is important to emphasize that this performance degradation is not caused by RAC code changes. No modifications were made to either of these services as part of the Read Access Control feature delivery. However, because adopting the RAC feature naturally encourages organizations to create and configure more Team Areas to partition artifact access, the total volume of team areas in an enterprise repository increases. Because these two preexisting foundation services are designed to traverse, serialize, and return complete process area hierarchies and user memberships, their execution latency is directly dependent on the total count of project areas and team areas in the repository.

To provide transparent guidance for architecture, capacity planning, and deployment design, the operational characteristics of these two services are detailed below.

1. Service: /processAreaHierarchiesWithApplicationContext

• Full Service Endpoint: /qm/service/com.ibm.team.process.internal.service.web.IProcessWebUIService/processAreaHierarchiesWithApplicationContext
• When Triggered: Executed automatically by the web client whenever a user loads or refreshes an ETM Project Area Dashboard.
• Behavior & Performance Impact: This service computes the complete process area hierarchy (all project areas and recursive team areas) associated with the application context. When an environment contains thousands of project areas and deeply nested team areas, this service experiences significant latency—taking up to ~1 minute (60 seconds) to return.
• Impact on End-User Experience: The initial dashboard loading spinner remains active for extended durations, degrading the perceived performance of project dashboards.

The single-user characterization chart below illustrates response time growth for processAreaHierarchiesWithApplicationContext as the team area count increases:

2. Service: /processAreasForUser

• Full Service Endpoint: /qm/service/com.ibm.team.process.internal.service.web.IProcessWebUIService/processAreasForUser
• When Triggered: Triggered during user navigating to artifact editors or detail views (such as opening a Test Plan, Test Case, or TCER).
• Behavior & Performance Impact: This service fetches and serializes all project areas and team areas to which the logged-in user is explicitly assigned. As a user's assigned team area count grows (e.g. users belonging to hundreds of team areas across large enterprise deployments), the payload generation and permission resolution time scale linearly.
• Impact on End-User Experience: Introduces incremental latency on individual artifact transitions and initial navigation clicks for users with broad team memberships.

The chart below shows single-user response times for processAreasForUser as a function of the user's assigned team area count:

Recommendations

The performance degradation in both services stems from recursive hierarchy traversal and process model serialization in the Jazz Foundation layer when handling massive team area sets.

Recommendations & Best Practices:

1. Team Area Sizing: Keep total team areas per project area within the tested sweet spot (500 to 1,000 team areas). Where larger counts are required, maintain user concurrency within the recommended limits (1,000 users for 2,000 team areas).
2. Principle of Least Privilege in Team Membership: Avoid assigning individual user accounts or service accounts globally to hundreds of team areas. Users should only be added to the specific team areas where they actively participate.
3. Dashboard Optimization: Avoid heavy process-area-dependent widgets on personal and project dashboards where large team hierarchies exist.

Additional Details on the Data Shape

To provide full context on the repository size used during testing, the breakdown of artifact distributions across project areas and overall database table volumes in the ETM repository is detailed below.

Repository Overview:

• Total Project Areas: 2,040
• Total Team Areas: 20,200
• Total Artifact Count in Repository Version Table: 216,162,299 artifacts

Artifact Counts in repository_version Table

Test Case Distribution by Project Area

Test Suite Distribution by Project Area

Test Case Execution Record (TCER) Distribution by Project Area

Appendix: Abbreviations and Terminology

References

• IBM Documentation: Read access control by using team areas in ETM