ETM Read Access Control (RAC) Performance Report

Introduction

For organizations who need team members—including external partners and suppliers—working together to manage the testing of complex systems in a coordinated manner while strictly complying with export control regulations such as the International Traffic in Arms Regulations (ITAR) and corporate data segmentation policies, Engineering Test Management (ETM) Read Access Control (RAC) using Team Areas is a major capability.

Historically, the only mechanism to isolate sensitive test artifacts was to create completely separate ETM project areas for different teams. However, this isolation created operational silos and hindered collaboration scenarios—such as maintaining a master Test Plan that links to or references restricted Test Cases managed by specific partner teams.

ETM Read Access Control provides granular visibility control within a single, non-configuration-enabled project area by associating read access permissions with Jazz Team Areas. Team members only see the test artifacts assigned to the team areas in which they hold membership (or child team areas thereof), while unauthorized users are prevented from viewing or querying restricted artifacts.

Business Value

Read access control using team areas provides significant enterprise and operational benefits:

• Export Regulation & ITAR Compliance: Enables strict compliance with export control laws and data segmentation mandates by enforcing strict data partitioning.
• Risk Mitigation: Prevents unauthorized read access and data leakage of confidential or classified test artifacts to suppliers or third-party collaborators.
• Unified Collaboration in Single Project Area: Eliminates project area duplication, enabling master test planning, unified dashboards, and cross-team test management within a single shared project area.
• Simplified Governance: Permissions are managed dynamically through team area membership, reducing administrative overhead.
• Full Auditability: Changes to read access control assignments on artifacts are tracked through standard artifact revision history.

Supported Test Artifacts

Read access control using team areas is supported across the core ETM artifact lifecycle:

• Test Plan
• Test Case
• Test Suite
• Test Script
• Test Case Execution Record (TCER)
• Test Suite Execution Record (TSER)
• Test Case Result
• Test Suite Result
• Test Data
• Keyword
• Attachment

Assignment of a team area for the purpose of read access control can be performed on individual test artifacts in their respective editors or in bulk directly from artifact browsing tables.

Architectural & Technical Overview

The ETM RAC feature leverages pre-existing Jazz Foundation (JAF) item-level read permission architecture:

• Read Access Policy: In-scope ETM items utilize a PROTECTED read access policy at the repository level.
• Read Access Context: Artifacts store a read access context reference pointing to the governing Team Area Item UUID. When an artifact is restricted to a Team Area, the Foundation access context is registered accordingly.
• SQL Access Right Filtering: When queries are executed against the relational database (Oracle / Db2), additional SQL filtering (incorporating Common Table Expressions / CTEs and subqueries) is dynamically applied to enforce process area and team area visibility for the executing user session.
• Transparent Security: Users without read permissions receive HTTP 404 (Not Found) responses upon direct URI access. In table browse views and query filters, unauthorized artifacts are filtered out of the result set without exposing metadata.
• Applicability Scope: RAC using team areas is supported on non-configuration-enabled (Non-CM) ETM project areas. Once RAC is enabled on a project area, it cannot be disabled, and the project area cannot subsequently be enabled for configuration management.

For additional feature documentation, refer to the IBM Documentation on ETM Read Access Control by Using Team Areas.

Standard Disclaimer

The information in this document is distributed AS IS. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer’s ability to evaluate and integrate them into the customer’s operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk. Any pointers in this publication to external Web sites are provided for convenience only and do not in any manner serve as an endorsement of these Web sites. Any performance data contained in this document was determined in a controlled environment, and therefore, the results that may be obtained in other operating environments may vary significantly. Users of this document should verify the applicable data for their specific environment.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multi-programming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

This testing was done as a way to evaluate and characterize the performance, scalability, and resource utilization of ETM Read Access Control under high concurrent user load and large data volumes. The results shown here should be evaluated as comparative performance data and operational guidance rather than an absolute benchmark for all custom deployments.

Key Findings

Performance and scalability characterization of ETM Read Access Control (RAC) by using Team Areas across large enterprise repositories (147M+ artifacts) established the following key conclusions:

• High Concurrency Scalability:
  • In typical enterprise project deployments with up to 500 team areas, RAC scales linearly up to 3,000 concurrent users with fast, sub-second response times and zero performance degradation.
  • Under standard mixed workloads up to 1,500 concurrent users, zero transactions breached the 5.0-second SLA threshold.

• Low Infrastructure Overhead:
  • The RAC SQL access-filtering logic introduces minimal CPU overhead on the hosting environment. At peak 1,500-user load, CPU utilization remained well below capacity—peaking at 12.4% on the Oracle 19c Database and 8.7% on the ETM Liberty Application Server.

• Team Area Sizing Boundaries:
  • Because RAC evaluates team hierarchies dynamically in SQL queries, high team area density per project introduces sizing boundaries:
    • 500 Team Areas: Recommended up to 3,000 concurrent users.
    • 1,000 Team Areas: Recommended up to 1,800 concurrent users (knee point at ~2,000 users).
    • 2,000 Team Areas: Recommended up to 1,000 concurrent users (elevated latency beyond 1,000 users).

• Identified Bottlenecks:
  • Initial project area dashboard loads invoke /processAreaHierarchiesWithApplicationContext, which can take up to ~1 minute in repositories with massive project and team area counts.
  • User permission resolution (/processAreasForUser) scales linearly with the number of team areas assigned to an individual user, warranting the practice of least privilege in user team assignments.

Test Methodology

This section details how the ETM RAC performance characterization and scale tests were conducted, including the repository data shape, environment topology, test scenarios, and workload execution methodology.

Data Shape & Repository Scale

Performance validation was executed against our largest ETM repository containing approximately 147M artifacts across 2,040 project areas and 20,200 team areas.

To stress-test the new RAC SQL filtering paths, hierarchy resolution logic, synthetic benchmark project areas and team area hierarchies were generated in two structured phases using the Java-based ETM Data Generator tool:

• Phase 1 — Project & Team Area Hierarchy Generation:
  • Created ~1,000 non-configuration ETM project areas.
  • Configured hierarchical team structures in each project: 10 top-level team areas, each having 1 sub-team area (20 team areas per project).
  • Assigned users across project and team areas with standard JazzUsers and JazzAdmin roles.

• Phase 2 — High-Volume RAC Project Areas:
  • Populated high-volume project areas generating ~250K to ~2.3M total artifacts with complex team area assignments.
  • Configured user populations from 500 up to 10,000 assigned users per project area.
  • Generated multiple artifact states and relationship links across Test Plans, Test Cases, Test Suites, Test Scripts, and Execution Records.

The following screenshot illustrates the team area hierarchy configuration established in the test project areas:

Workload & Test Scenarios

The performance workload was designed to simulate realistic end-user navigation patterns, browsing actions, and complex filtering operations across RAC-enabled repositories. The core scenario suite included:

1. Authentication & Dashboard Navigation:
   • RQM_Login: User authentication and initial session handshake.
   • RQM_Dashboard: Initial project dashboard rendering and widget data retrieval.
2. Test Plan & Artifact Browsing:
   • QM_Browse_TestPlans: Loading the paginated Test Plan table.
   • QM_Open_Random_TestPlan: Opening a full Test Plan editor.
   • QM_Open_TP_TC_View: Browsing Test Cases associated with a selected Test Plan.
   • QM_Open_TP_TS_View: Browsing Test Suites associated with a selected Test Plan.
   • QM_Open_TP_TCER_View: Browsing Test Case Execution Records (TCERs) associated with a selected Test Plan.
   • QM_Open_TCs_List_View: Browsing the main Test Cases table view.
   • QM_Open_TCERs_List_View: Browsing the main Test Case Execution Records (TCERs) table view.
3. Direct Artifact Opening:
   • QM_Open_Random_TestCase, QM_Open_Random_TestSuite, QM_Open_Random_TCER.
4. TCER Filter Operations:
   • Iterations
   • Last Result Passed
   • Last Result Failed
   • Blocked / Error
   • Owners
   • Owners + Weights
   • Owners Exclude Iterations
   • Test Environment + Owners
5. Test Case Filter Operations:
   • Categories
   • Categories Excluding
   • Categories (Some Unassigned)
   • Draft Status
   • Created By
   • Owners
   • Categories + Test Plan Script Steps
   • Categories + Test Suites + Test Scripts
   • Categories + Weight

Load Testing Methodology

Multi-user workload simulation was executed using Rational Performance Tester (RPT):

• Multi-user load schedules were built to execute mixed-scenario workloads with realistic pacing and think time.
• Multi-tier scale runs were conducted from 10 concurrent users up to 1,500 concurrent users (with dedicated scaling capacity evaluations up to 3,000 users).
• Workloads were executed side-by-side against both Non-CM (RAC) and CM-Aware project areas under identical hardware and database configurations to isolate and quantify the RAC access-filtering overhead.

The following screenshot shows the RPT test schedule structure:

Monitoring Tools

Comprehensive monitoring was active throughout all benchmark windows:

• NMON: Continuous recording of CPU, memory utilization, disk I/O, and network throughput across all physical nodes.
• Oracle AWR (Automatic Workload Repository): Profiling database execution time, RAC Common Table Expression (CTE) query overhead, buffer cache hits, and SQL execution plans.
• Java Diagnostics: Thread dumps, javacores, and queryStats instrumentation to detect internal lock contention or JVM memory pressure.
• RPT Report: Aggregating transaction throughput, response times, and error rates.

Test Environment Details

The performance tests were conducted on an enterprise four-server topology deployed on bare-metal physical servers.

Performance Test Results

This section presents the performance test findings, capacity thresholds, comparative metrics between Non-CM (RAC) and CM-Aware project areas, and server resource utilization under load up to 1,500 concurrent users.

Summary of Load Test Results

• Stable Scalability to 1,500 Users: ETM Non-CM project areas with Read Access Control enabled scale reliably up to 1,500 concurrent users with zero response time threshold breaches (> 5.0 seconds SLA).
• Minimal Server Resource Overhead: Server CPU utilization remained exceptionally low on both tiers at 1,500 concurrent users—peaking at 12.4% on the Oracle Database server and 8.7% on the ETM Application server.
• Fast Navigation & TCER Filtering in Non-CM: Browsing Test Plans (~236 ms), opening TCER views (~328 ms), and executing TCER filters (285–307 ms) demonstrated sub-second response times across all load levels.
• Predictable Scaling Profile: Response times for navigation, dashboards, and filtering remained flat from 100U to 1,500U, demonstrating strong query execution stability.

Load Testing with Varying Team Area Counts (Capacity Limits)

To establish operational sizing guidelines, dedicated load testing was conducted with varying numbers of team areas (500, 1,000, and 2,000 team areas) under increasing concurrent user counts. Because RAC SQL filters evaluate team hierarchy paths and memberships during query execution, team area density directly influences database query planning and memory structures.

The testing revealed clear scaling thresholds:

• 500 Team Areas:
  • Tested up to 3,000 concurrent users.
  • Response times remained flat and fast across the entire load spectrum.
  • No measurable performance degradation was observed.
  • Recommended Operational Limit: Up to 3,000 concurrent users.

• 1,000 Team Areas:
  • Performance remained stable and well within acceptable thresholds up to approximately 1,800 concurrent users.
  • As load approached 2,000 concurrent users, response times began to experience exponential growth due to increased database CTE resolution overhead and lock queues.
  • Recommended Operational Limit: Up to 1,800 concurrent users.

• 2,000 Team Areas:
  • Performance degradation became noticeable starting at 1,000 concurrent users.
  • Elevated response times and query latency were consistently observed beyond 1,000 users as the high volume of team areas magnified the cost of hierarchy evaluation per query.
  • Recommended Operational Limit: Up to 1,000 concurrent users.

Summary of Sizing Guidelines by Team Area Scale

Server CPU Utilization

Resource monitoring confirmed that neither the ETM application server nor the Oracle database server experienced CPU exhaustion during the 1,500 user scale test. Both Non-CM (RAC) and CM-Aware workloads exhibited similar, moderate CPU growth profiles.

The charts below show the CPU utilization curves on the ETM Application Server and Database Server as user load scaled from 10 to 1,500 concurrent users:

Detailed Response Time Analysis: Non-CM (RAC) vs CM-Aware

The multi-user benchmark measured 31 distinct user transactions and queries across both project area modes.

1. Navigation & View Operations

2. TCER Filter Operations

TCER filtering demonstrated high efficiency under RAC in Non-CM project areas, consistently executing in 280 ms to 380 ms. In CM-Aware project areas, TCER queries executed in the 1.7 s to 2.0 s range due to configuration-context resolution overhead across versioned configurations.

3. Test Case Filter Operations

In Test Case filtering, queries involving negative category filters (Categories Excluding) or unassigned status (Draft) in Non-CM RAC project areas require broader table scans and multi-join CTE evaluations across the large unversioned artifact repository. These operations took ~4.2–4.3 s at low load (cold cache) and stabilized at ~3.0–3.1 s at 1,500 users. In CM-Aware project areas, the versioned component scoping allows these queries to execute faster (~150 ms).

Performance Limitations

During rigorous performance and single-user scale profiling, two existing Jazz Foundation Web UI services were identified as performance bottlenecks when the total number of project areas, team areas, and user memberships in an environment becomes very large.

Important Technical Context: It is important to emphasize that this performance degradation is not caused by RAC code changes. No modifications were made to either of these services as part of the Read Access Control feature delivery. However, because adopting the RAC feature naturally encourages organizations to create and configure more Team Areas to partition artifact access, the total volume of team areas in an enterprise repository increases. Because these two preexisting foundation services are designed to traverse, serialize, and return complete process area hierarchies and user memberships, their execution latency is directly dependent on the total count of project areas and team areas in the repository.

To provide transparent guidance for architecture, capacity planning, and deployment design, the operational characteristics of these two services are detailed below.

1. Service: /processAreaHierarchiesWithApplicationContext

• Full Service Endpoint: /qm/service/com.ibm.team.process.internal.service.web.IProcessWebUIService/processAreaHierarchiesWithApplicationContext
• When Triggered: Executed automatically by the web client whenever a user loads or refreshes an ETM Project Area Dashboard.
• Behavior & Performance Impact: This service computes the complete process area hierarchy (all project areas and recursive team areas) associated with the application context. When an environment contains thousands of project areas and deeply nested team areas, this service experiences significant latency—taking up to ~1 minute (60 seconds) to return.
• Impact on End-User Experience: The initial dashboard loading spinner remains active for extended durations, degrading the perceived performance of project dashboards.

The single-user characterization chart below illustrates response time growth for processAreaHierarchiesWithApplicationContext as the team area count increases:

2. Service: /processAreasForUser

• Full Service Endpoint: /qm/service/com.ibm.team.process.internal.service.web.IProcessWebUIService/processAreasForUser
• When Triggered: Triggered during user navigating to artifact editors or detail views (such as opening a Test Plan, Test Case, or TCER).
• Behavior & Performance Impact: This service fetches and serializes all project areas and team areas to which the logged-in user is explicitly assigned. As a user's assigned team area count grows (e.g. users belonging to hundreds of team areas across large enterprise deployments), the payload generation and permission resolution time scale linearly.
• Impact on End-User Experience: Introduces incremental latency on individual artifact transitions and initial navigation clicks for users with broad team memberships.

The chart below shows single-user response times for processAreasForUser as a function of the user's assigned team area count:

Recommendations

The performance degradation in both services stems from recursive hierarchy traversal and process model serialization in the Jazz Foundation layer when handling massive team area sets.

Recommendations & Best Practices:

1. Team Area Sizing: Keep total team areas per project area within the tested sweet spot (500 to 1,000 team areas). Where larger counts are required, maintain user concurrency within the recommended limits (1,000 users for 2,000 team areas).
2. Principle of Least Privilege in Team Membership: Avoid assigning individual user accounts or service accounts globally to hundreds of team areas. Users should only be added to the specific team areas where they actively participate.
3. Dashboard Optimization: Avoid heavy process-area-dependent widgets on personal and project dashboards where large team hierarchies exist.

Additional Details on the Data Shape

To provide full context on the repository size used during testing, the breakdown of artifact distributions across project areas and overall database table volumes in the ETM repository is detailed below.

Repository Overview:

• Total Project Areas: 2,040
• Total Team Areas: 20,200
• Total Artifact Count in Repository Version Table: 216,162,299 artifacts

Artifact Counts in repository_version Table

Test Case Distribution by Project Area

Test Suite Distribution by Project Area

Test Case Execution Record (TCER) Distribution by Project Area

Appendix: Abbreviations and Terminology

References

• IBM Documentation: Read access control by using team areas in ETM