7.0.1 Performance Report: IBM Engineering Test Management

Introduction

[1]: versioned test artifact counts are based on the data generation utility which collects the sum from each artifact type visible in the UI.

Changes to the 7.0.1 ETM performance test

• Topology: replaced the IBM WebSphere Application Servers with Embedded WebSphere Liberty Servers, replaced IBM DB2 Server with Oracle
• Datashape: added multiple ETM components to the same project area, mixed different component sizes
• Repository size: increased the total count of versioned test artifacts to 15 million

The test in this report does not cover the evaluation of performance under Global Configuration, nor capacity/scalability with regard to increasing the number of streams or baselines for any given ETM component. The integration scenarios including cross-product links are not tested in this release.

While the report provides a Summary of results, detailed information is covered in various sections. Use the page content menu on the upper-right section of the report to navigate. We may provide raw results upon requests.

Disclaimer

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multi-programming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

This testing was done as a way to compare and characterize the differences in performance between different versions of the product. The results shown here should thus be looked at as a comparison of the contrasting performance between different versions, and not as an absolute benchmark of performance.

Summary of results

• When we increased the size of the repository from 10 million ^[2] to 15 million total number of versioned test artifacts, or by 50%, for the same extra large-sized component:
  • The general performance only slightly degrades, with an average response time for all pages increased by 20% approximately.
  • 15 out of 128, or 11.7%, of the pages under test have an increase of at least 1 second, but no more than 3 seconds, with this level of repository size growth.
  • The top 15 pages with the most increases in response time include various read operations in a stream, such as Browse Test Case Results, Browse Test Case Execution Records, and pages to search/filter a type of test artifact, due to the largest test artifact counts for these pages, as shown in chart below. Amongst them, many of the searchTestArtifact pages have a chance to be more optimal if they had been tuned via SQL profiling approach outlined in Test load based tunings.

    

    .
  • The server resource utilizations are barely increased: the CPU utilizations on all servers are still low, disk %busy on ETM WebSphere Liberty Server remains moderate. We noticed the Oracle disk busyness is about the same during the run phase regardless of repository size, with an average of 88% which includes handling multiple background tasks by the server post migration. These background tasks are estimated to introduce 60% extra disk utilizations.
  • For the page-by-page performance result for each test scenario, see this section: Detailed performance results.
  • [2]: the 10 million -repository performance number is based on ETM 7.0 GA data; however we estimated that the performance impact between ETM 7.0 and 7.0.1 is minimal.
• When we compared the performance of a much smaller component (size of 500 thousand versioned test artifacts) versus the extra large component (size of 10 million versioned test artifacts):
  • The result shows the average response time for all pages have increased as the component size increases.
  • 28 out of 128, or 22.8%, of the pages under test have an increase of at least 1 second, with the component size grows 20 times bigger; 9 pages have increased more than 3 seconds.
  • The larger counts in an artifact view, especially in baselines, the more impact is by the flat growth of their component size, which can be seen from the list below showing top pages that are mostly sensitive to the component size increase:

    

  • The server resource utilizations are the same or slightly increased.

Topology, network, and data volume

Data volume and shape

• ETM Database size on disk ~ 775 GB

Versioned test artifacts distribution by components

The test artifact maintains the following associations:

• each test plan is linked to 600 unique test cases, and 600 unique test scripts
• each test suite contains 15 test cases in smaller-sized component, and 500 test cases in the extra large component, respectively

Streams and baselines

EWM artifacts in the applications

Topology and network

The specifications of machines under test are listed in the table below. Server tuning details listed in Appendix - Key configuration parameters

N/A: Not applicable. vCores = Cores with hyperthreading

Network connectivity

All server machines and test clients are located on the same subnet. The LAN has 1000 Mbps of maximum bandwidth and less than 0.3 ms latency in ping.

Performance load

Rational Performance Tester (RPT) is used to simulate this load created using the web client. Each virtual user completes a random use case from a set of available use cases. Each use case is implemented as a RPT script; and these RPT scripts are organized by pages representing user actions.

The performance work load is role based as each of the areas defined under sequence of actions which are separated into individual user groups within an RPT schedule, see User roles, test cases and workload characterization for detailed description for each test scenario.

User roles, test cases and workload characterization

b. Test Cases

Response time comparison

Detailed performance results

Average page response time breakdown

For all charts, each column represents the time for page loading (or page response time) in milliseconds, for which value smaller is better.

Browse Test Plans & Test Cases

repository size increased by 50%	component size increased by 2,000%
* Note: this page can be variable; however the 90th percentile of this page is under 1 second in both cases.

Browse Test Scripts

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Test Execution Record Browsing

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Create Defect

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Create Test Plan

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Create Test Case

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Create Test Script

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Edit Test Case

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Edit Test Environment

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Edit Test Plan

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Edit Test Script

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Test Execution

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Simple Test Plan Copy

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Bulk Edit of Test Cases

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

View Dashboard

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Browse Test Script In Baseline

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Browse Test Case Execution Record In Baseline

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Browse Test Plans And Test Cases In Baseline

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Defect Search

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

Defect Modify

repository size increased by 50%	component size increased by 2,000%

Back to Test Cases & workload characterization

RPT script executions

	Test scenario	Average execution counts per load test
	Execute the TER (Test Execution For 4 Steps)	1,478
	Browse TER	864
	Search Defect	650
	View Dashboard	508
	Browse Test Script	460
	Edit Test Script	245
	Create Defect	222
	Edit Test Case	170
	Edit Test Plan	163
	Modify Defect	150
	Create Test Script	119
	Browse Test Plans And Test Cases	110
	Refresh Dashboard	98
	Edit Test Environment	91
	Create Test Case	64
	Browse Test Script In Baseline	23
	Browse TER In Baseline	23
	Browse Test Plans And Test Cases In Baseline	17
	Full Text Search and Open Test Suite	0
	Bulk Edit of Test Cases	17
	Create Test Plan	2
	Simple Test Plan Copy	2

RPT network transfer comparison

Resource utilization

Database Server resource utilization - ETM 7.0.1 15 million repository - 1000users performance load large scale:

	extra large component vs smaller component
CPU utilization		Disk utilization
Memory		Disk transfer

WebSphere Liberty Server (ETM) resource utilization - ETM 7.0.1 15 million repository - 1000users performance load large scale:

	extra large component vs smaller component
CPU utilization		Disk utilization

Garbage collection - JVM GC Chart

Verbose garbage collection is enabled to log the GC activities. These GC logs showed very little variation between runs. There is also no discernible difference between versions. Below is one example of the output from the GC log for each application.

WebSphere Liberty Server JVM Garbage Collections

Test load based tunings

Apply additional indexes based on the test load

These pages could be the most expensive ones in our performance test, taking more than 10 seconds to load for instance. When we looked at the queries from within the database, we determined to use additional indexes to help improve both of their response times as well as the memory usage. The indexes were created on the ETM database schema as below

--function-based indexes for using name column for filtering and searching
CREATE INDEX "QM_USER"."NAMECOL_UPPER_TER" ON "QM_USER"."EXECUTION_EXECUTION_RECORD" (UPPER("NAME"));
CREATE INDEX "QM_USER"."NAMECOL_UPPER_TCER" ON "QM_USER"."EXECUTION_EXECUTION_TASK_RESLT" (UPPER("NAME"));
CREATE INDEX "QM_USER"."NAMECOL_UPPER_TS" on "QM_USER"."PLANNING_EXECUTION_SCRIPT" (UPPER(NAME));
CREATE INDEX "QM_USER"."NAMECOL_UPPER_CATG" on "QM_USER"."PLANNING_CATEGORY" (UPPER(NAME));
CREATE INDEX "QM_USER"."NAMECOL_UPPER_TC" on "QM_USER"."PLANNING_TEST_CASE" (UPPER(NAME));
CREATE INDEX "QM_USER"."NAMECOL_UPPER_TSU" on "QM_USER"."PLANNING_TEST_SUITE" (UPPER(NAME));
CREATE INDEX "QM_USER"."NAMECOL_UPPER_CTRR" on "QM_USER"."REPOSITORY_CONTRIBUTOR_RECORD" (UPPER(NAME));
CREATE INDEX "QM_USER"."USERID_UPPER_CTRR" on "QM_USER"."REPOSITORY_CONTRBTR_RCRD_SR_DS" (UPPER(USER_ID));

--composite indexes to improve large views such as execution record, and test results
CREATE INDEX "QM_USER"."TER_JZ_ARC_MOD_PRJ_ITM_ST" ON "QM_USER"."EXECUTION_EXECUTION_RECORD" ("JZ_DISCRIMINATOR", "ARCHIVED", "MODIFIED", "PROJECT_AREA_ITEM_ID", "ITEM_ID", "STATE_ID"); 
CREATE INDEX "QM_USER"."TCER_JZ_ARC_ET_PRJ_ITM_ST" ON "QM_USER"."EXECUTION_EXECUTION_TASK_RESLT" ("JZ_DISCRIMINATOR", "ARCHIVED", "END_TIME", "PROJECT_AREA_ITEM_ID", "ITEM_ID", "STATE_ID"); 
CREATE INDEX "QM_USER"."JZ_ARC_PRJ_SNPS_ITM_ST" ON "QM_USER"."PLANNING_VERSIONABLE_ITEM" ("JZ_DISCRIMINATOR", "ARCHIVED", "PROJECT_AREA_ITEM_ID", "SNAPSHOT", "ITEM_ID", "STATE_ID"); 

--composite index for process
CREATE INDEX "QM_USER"."PRJITMID_ITMID" ON "QM_USER"."PROCESS_PROCESS_AREA" ("PROJECT_AREA_ITEM_ID", "ITEM_ID");

Apply SQL profile

• emulate the searching and filtering scenarios, via web browser or automation
• collect theses queries (this can be done either from tracking the hotspot queries from the Oracle Performance Hub, or found in the AWR report)
• create tuning tasks, and execute tuning tasks
• review tuning suggestion, make sure the function-based indexes are being used by the EXPLAIN plan with which performance is improved
• accept tuning task that applies the SQL profile

In ETM 7.0.1, we found that the option to allow function-based indexes through SQL profiling may disappear due to an index "RPSTR_VRSNSTDBDSCRNTTMCRTDCNFG" presented under the ETM user schema. After dropping this index, the EXPLAIN PLAN from SQL profile will become desired. Based on the performance payload test and extended longer loads, dropping this particular index in the ETM user schema does not seem to introduce negative performance impact to the application.

Oracle tuning parameters

####enable automatic memory management (AMM)
#original memory_max_target and memory_target are both 0;
#original sga_target=28800M and sga_max_size=40G;
#may need to increase the memory settings on the system first,  run as root: mount -t tmpfs shmfs -o size=56g /dev/shm
alter system set memory_max_target=52000M scope=spfile;
alter system set memory_target=52000M scope=spfile;
alter system set sga_max_size=0 scope=spfile;
alter system set sga_target=0 scope=spfile;
alter system set pga_aggregate_limit=0 scope=spfile;
alter system set pga_aggregate_target=0 scope=spfile;
#rebounce db. note that in the test we also secured the sizes of sga_max_size and db_cache_size to match the high water mark observed prior to tuning
alter system set sga_max_size=40G scope=spfile;
alter system set db_cache_size=24G scope=spfile;
#rebounce db

####disable adaptive plan
alter system set optimizer_adaptive_plans = false #default true
#_optimizer_use_feedback= false #not available in 12c

####these settings can help enable auto parallelism in SQL execution. however for high payload, only turn these on when you have enough disk capacity
#alter system set parallel_degree_policy=auto scope=spfile; #default manual
#alter system set filesystemio_options=setall scope=spfile; #default none
#rebounce db 

Appendix - Key configuration parameters

 -Xgcpolicy:gencon -Xmx16g -Xms16g -Xmn4g -Xcompressedrefs -Xgc:preferredHeapBase=0x100000000
 -verbose:gc -Xverbosegclog:gc.log -XX:MaxDirectMemorySize=1G

About the authors:

Related topics: Engineering Test Management (ETM) Adapter Deployment Guide,DOORS Next performance guide,Performance datasheets

Questions and comments:

• What other performance information would you like to see here?
• Do you have performance scenarios to share?
• Do you have scenarios that are not addressed in documentation?
• Where are you having problems in performance?

Warning: Can't find topic Deployment.PerformanceDatasheetReaderComments