E
dit
A
ttach
P
rintable
r6 - 2020-12-02 - 18:33:11 - Main.hhuo
You are here:
TWiki
>
Deployment Web
>
DeploymentPlanningAndDesign
>
PerformanceDatasheetsAndSizingGuidelines
>
702PerformanceReportIBMEngineeringTestManagement
<div id="header-title" style="padding: 10px 15px; border-width:1px; border-style:solid; border-color:#FFD28C; background-image: url(<nop>https://jazz.net/wiki/pub/Deployment/WebPreferences/TLASE.jpg); background-size: cover; font-size:120%"> <!-- * Set ALLOWTOPICCHANGE = Main.TWikiDeploymentDatasheetsAuthorsGroup, Main.TWikiAdminGroup, Main.GrantCovell --> ---+!! <img src="https://jazz.net/wiki/pub/Deployment/WebPreferences/new.png" alt="new.png" width="50" height="50" align="right">IBM Engineering Test Management (ETM) 7.0.2: performance and scalability updates</br> %DKGRAY% Author: [[Main.HongyanHuo][Hongyan Huo]]</br> Last updated: Nov 20, 2020</br> Build basis: IBM Engineering Test Management 7.0.2 %ENDCOLOR%</div></sticky> <!-- Page contents top of page on right hand side in box --> <sticky><div style="float:right; border-width:1px; border-style:solid; border-color:#DFDFDF; background-color:#F6F6F6; margin:0 0 15px 15px; padding: 0 15px 0 15px;"> %TOC{title="Page contents"}% </div></sticky> <sticky><div style="margin:15px;"></sticky> ---+++!! Standard disclaimer %INCLUDE{"PerformanceDatasheetDisclaimerEndToEnd"}% ---++ Updates from [[ELMLargeScaleAndPerformanceReportRelease701][previous report]]: This report provides an update to IBM Engineering Test Management (ETM) 7.0.2 performance and scalability test, from the [[https://jazz.net/wiki/bin/view/Deployment/ELMLargeScaleAndPerformanceReportRelease701][7.0.1 release]]. ---++++Datashape In this release, the datashape has been evolved as outlined below * Number of components: total number of components is increased from 11 to 2511 * Repository size<font size=1>[1]</font>: increased from 15 million to 20 million total artifacts, or by 33% * ETM Database size on disk has grown from 775GB to 1171GB * Component size<font size=1>[1]</font>: * added small (<1K), medium (5K), and large (50K) sizes * keeping extra large (500K) and extra-extra-large (10M) sizes <i><font size=1>[1]</font>: measured by total number of versioned artifacts generated by the [[https://jazz.net/wiki/bin/view/Main/RQMDatagen][Data Generation Tool]] which collects the sum from each artifact type visible in the UI.</i><p> The size and number of different components are summarized in the table below %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="versioned test artifact counts by component size: " tableborder="2" tableframe="border" tablerules="none"}% |Counts|Extra-extra large component(10M) |Extra large component(500K) |Large(50K) |Medium(5K) | Small(<1K)| Sum in Repo | |test plans|1,681|50|6|4|1|6,281| |test cases|1,000,207|30,000|3,000|400|20|1,670,207| |test scripts|1,000,209|30,000|3,000|400|20|1,670,209| |test suites|100,020|3,000|300|40|10|183,020| |test case execution records|4,000,800|120,000|12,000|1,200|40|6,420,800| |test case results|4,000,921|360,000|36,000|2,400|80|10,640,921| |test suite results|500,100|15,000|1,500|160|20|837,100| |test execution schedules|47,467|1,424|500|200|20|216,707| |test phases and test environments|31,636|800|92|63|20|112,586| |build definitions and build records|33,533|1,006|120|70|25|131,093| |Total # of artifacts per component|10,716,574|561,280|56,518|4,937|256|21,888,924| |Total # of components in repository|1|10|50|450|2000|2,511| |Newly added in 702|N|N|Y|Y|Y|-| ---++++Configuration test coverage The performance and load tests of small, medium, and large sized components are evaluated under Global Configuration (GC) for this release. The GC component is consisted of all 2500 ETM components. The extra and extra-extra large sized components remain to be tested against their local ETM configurations including both streams and baselines for regression purpose. ---++++Disclaimer The extra-extra-large sized component was created initially to expediate the growth of the repository without consideration of artifact versions (i.e. the artifacts were generated in an opt-out environment prior to the enablement of Configuration Management feature). The test against this particular component was initially experimental during 7.0, but it continuously serves as a way to regression test the performance of IBM IBM Engineering Test Management (ETM). The results using this component can represent edge cases due to its huge size, thus the measurement for the performance of this component shown in the report should be looked at and analyzed as a comparison of the performance of different releases in the context of datashape. It should not be deemed as a performance benchmark. ---++ Summary of regression test results We continue to find the following performance characteristics as we keep increasing the size of the repository ---++++ Component performance vs. component size For any given repository size, generally: * The smaller a component, the faster page response times or better performance for that component. Under the 20 million repository, for components that have a size ranged from small to large, 98% of the individual page response times are under 2 seconds (with the exception of Dashboard loading being 4 seconds, and Saving a test plan may take 5 seconds due to large amount of iterations defined); whereas 11 out of 120 pages are exceeding 5 seconds for the extra-large component, most of these slower pages are browsing/searching a given artifact type that has a larger total count, including searching/filtering test case results (9 seconds) and test case records (5.5 seconds) in the default views within a stream. The detailed analysis of the component performance as a function of component size can be found in [[ELMLargeScaleAndPerformanceReportRelease701][ETM 7.0.1 Performance Report]]. Its worthy to note that since the component size can vary substantially, each of the individual test load was only focused on homogeneously sized components. Carrying out test loads this way not only leads to statistically normalization of page response times, but also allows servers to warmup thoroughly and database become more optimal. ---++++ Component performance vs. growing repository size * The general performance of a given component<font size=1>[2]</font> degrades as the repository size grows. Although the measurements based on extre-extra-large sized component stress tested some edge cases, we observed a projection of a slight degradation when comparing between the 15-million repository and the 10-million repository, yet a minor to moderate degradation in comparing the 20-million repository to the 15-million repository. The degradation degree for 500K component is still minor under 20-million repository (average page response time for all pages degraded by less than 10%, comparing to 15-million repository). * Within a component context, the pages that are more sensitive to the size of the entire repository include those embracing the largest test artifact counts, for instances, in our test environment, page Browse Test Case Execution Records (total count of 4 million in a single component, or 30% of total artifact count in the repository), Browse Test Case Results, and the relevant pages to search/filter the test artifact. For a detailed analysis of the component performance as a function of repository size, please refer to [[ELMLargeScaleAndPerformanceReportRelease701][ETM 7.0.1 Performance Report]]. <i><font size=1>[2]</font>: component size did not grow prior to the performance load test. </i> ---++ Whats not in scope of test for this release Like all previous releases, scalability that may be impacted by any of the following dimensions are not covered in 7.0.2 tests - * The growth of number of baselines for a given component or stream * The growth of number of versions for a given configuration * The growth of number of states for a given artifact ---++ References: * [[https://jazz.net/wiki/bin/view/Deployment/ELMLargeScaleAndPerformanceReportRelease701#Topology_and_network][Topology and hardware]] * [[https://jazz.net/wiki/bin/view/Deployment/ELMLargeScaleAndPerformanceReportRelease701#Appendix_Key_configuration_param][Application server and database version and configuration under test]] * [[https://jazz.net/wiki/bin/view/Deployment/ELMLargeScaleAndPerformanceReportRelease701#Performance_load][Performance load]] * [[https://jazz.net/wiki/bin/view/Deployment/ELMLargeScaleAndPerformanceReportRelease701#Detailed_performance_results][Page by page performance results in 7.0.1]] * [[https://jazz.net/wiki/bin/view/Deployment/ELMLargeScaleAndPerformanceReportRelease701#Test_load_based_tunings][Test load based tunings]] ---++++!! Related topics: [[ELMLargeScaleAndPerformanceReportRelease701][ETM 7.0.1 Performance Report]], [[https://jazz.net/wiki/bin/view/Main/ETMAdapterDeploymentGuide#Configuration][Engineering Test Management (ETM) Adapter Deployment Guide]], [[https://jazz.net/wiki/bin/view/Deployment/RequirementsManagement70Performance][DOORS Next performance guide]] -------------------- ---+++++!! Questions and comments: * What other performance information would you like to see here? * Do you have performance scenarios to share? * Do you have scenarios that are not addressed in documentation? * Where are you having problems in performance? %COMMENT{type="below" target="PerformanceDatasheetReaderComments" button="Submit"}% %INCLUDE{"PerformanceDatasheetReaderComments"}% <sticky></div></sticky>
E
dit
|
A
ttach
|
P
rintable
|
V
iew topic
|
Backlinks:
We
b
,
A
l
l Webs
|
H
istory
: r6
<
r5
<
r4
<
r3
<
r2
|
M
ore topic actions
Deployment
Deployment web
Planning and design
Installing and upgrading
Migrating and evolving
Integrating
Administering
Monitoring
Troubleshooting
Community information and contribution guidelines
Create new topic
Topic list
Search
Advanced search
Notify
RSS
Atom
Changes
Statistics
Web preferences
NOTE: Please use the Sandbox web for testing
Status icon key:
To do
Under construction
New
Updated
Constant change
None - stable page
Smaller versions of status icons for inline text:
Copyright © by IBM and non-IBM contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our
Terms of Use.
Please read the following
disclaimer
.
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more
here
.