Edit
Attach
P
rintable
r15 - 2020-06-22 - 22:47:33 - Main.hhuo
You are here:
TWiki
>
Deployment Web
>
DeploymentPlanningAndDesign
>
PerformanceDatasheetsAndSizingGuidelines
>
ELMLargeScaleAndPerformanceReportRelease701
<div id="header-title" style="padding: 10px 15px; border-width:1px; border-style:solid; border-color:#FFD28C; background-image: url(<nop>https://jazz.net/wiki/pub/Deployment/WebPreferences/TLASE.jpg); background-size: cover; font-size:120%"> <!-- * Set ALLOWTOPICCHANGE = Main.TWikiDeploymentDatasheetsAuthorsGroup, Main.TWikiAdminGroup, Main.GrantCovell --> ---+!! <img src="https://jazz.net/wiki/pub/Deployment/WebPreferences/uc.png" alt="new.png" width="50" height="50" align="right">7.0.1 Performance Report: IBM Engineering Test Management (DRAFT)</br> %DKGRAY% Authors: [[Main.HongyanHuo][Hongyan Huo]]</br> Last updated: Jun 18, 2020</br> Build basis: IBM Engineering Test Management 7.0.1 %ENDCOLOR%</div></sticky> <!-- Page contents top of page on right hand side in box --> <sticky><div style="float:right; border-width:1px; border-style:solid; border-color:#DFDFDF; background-color:#F6F6F6; margin:0 0 15px 15px; padding: 0 15px 0 15px;"> %TOC{title="Page contents"}% </div></sticky> <sticky><div style="margin:15px;"></sticky> ---++ Introduction This article provides the large scale performance report of IBM Engineering Test Management <b> (ETM) </b> 7.0.1. In this report, we will compare the performance of a very large ETM component impacted by the repository size, and the ETM performance impacted by the component size in the same repository, repectively. The performance load is focused on a wide range of user operations in the <b>streams</b> (for instance, viewing, searching, adding, and modifying an artifact), incorporating a small portion of basic user navigations for <b>baselines</b>. The repository under test is comprised of <i>15 million versioned test artifacts</i><sup>1</sup> in eleven components migrated from 6.0.6, out of which one single extremely large-sized component contains ten million artfacts, while the remaining artifacts are evenly spead across ten smaller components. During the migration process, The ETM database were partitioned for scalability (refer to [[https://jazz.net/help-dev/clm/index.jsp?topic=%2Fcom.ibm.jazz.install.doc%2Ftopics%2Froadmap_clm_upgrade.html][upgrade guide]], [[https://jazz.net/pub/new-noteworthy/foundation/6.0.6.1/6.0.6.1/index.html#10][release new and noteworthy]], and [[https://www.ibm.com/support/knowledgecenter/SSYMRC_6.0.6.1/com.ibm.jazz.install.doc/topics/r_repotools_partitioning.html][the knowledge center]]for details for partitioning). <sup>1</sup><i><b> versioned artifact counts</b></i> are based on the [[https://jazz.net/wiki/bin/view/Main/RQMDatagen][data generation]] utility which collects the sum from each artifact type visible in the UI. ---++++ Changes to the 7.0.1 ETM performance test * [[#Topology_and_network][Topology]]: replaced the IBM !WebSphere Application Servers with Embedded !WebSphere Liberty Servers, replaced IBM DB2 Server with Oracle * [[#Artifacts_distribution_by_compon][Datashape]]: added multiple ETM components to the same project area, mixed different component sizes * Repository size: increased the total count of versioned artifacts to 15 millions The test in this report does not cover the evaluation of performance under Global Configuration, nor capacity/scalability with regard to increasing the number of streams or baselines for any given ETM component. While the report provides a [[#Summary_of_results][Summary of results]], detailed information is covered in various sections. Use the <b> page content </b> menu on the upper-right section of the report to navigate. We may provide raw results upon requests. ---+++!! Disclaimer <font face="Georgia" size=2> %INCLUDE{"PerformanceDatasheetDisclaimerEndToEnd"}% </font> ---++ Summary of results On top of the product scalablity improvement, we applied extra optimization by taking the adventage of database optimization approaches (refer to section [[#Test_load_based_tunings][performance tunings]] for details), given the unique design of the load which is keyword searching and sorting intensive throughout the scenarios. With them, the concurrency load was carried smoothly and maintained stable during of the entire course of testing. We observed the following performance characteristics * When we increased the size of the repository from 10 million<sup>2</sup> to 15 million total number of versioned artifacts, or by 50%, for the same large-sized component: * The general performance only slightly degrade, with an average response time for all pages increased by 20% approximately * 15 out of 128, or 11.7%, of the pages under test have an increase of at least 1 second, but no more than 3 seconds, with this level of repository size growth * The top 15 pages with the most increases in response time include various read operations in a stream, such as <b><i>Browse Test Case Results</i></b>, <b><i>Browse Test Case Execution Records</i></b>, and pages to search/filter a type of artifact, due to the largest artifact counts for these pages, as shown in chart below. Amongst them, many of the search<i>Artifact</i> pages have chance to be more optimal if they had been tuned via SQL profiling approach outlined in [[#Test_load_based_tunings][performance tunings]]. <p> <a title="click to enlarge image" href="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/topPageIncreases_10Mcmpnnt_15MrepoVS10Mrepo.png"> <img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/topPageIncreases_10Mcmpnnt_15MrepoVS10Mrepo_small.png" /> </a> </p>. * The server resource utilizations are barely increased: the CPU utilizations on all servers are still low, disk %busy on ETM !WebSphere Liberty Server remains moderate. We noticed the Oracle disk busyness% is about the same during the run phase regardless of repository size, with an average of 88% which includes handling multiple underground tasks by the server post migration. These background tasks are estimated to introduce 60% extra disk utilizations. * For the page-by-page performance result for each test scenario, see this section: [[#DetailedResults][Detailed performance results]]. * <sup>2</sup> the 10 million -repository performance number is based on ETM 7.0GA data; however we estimated that the performance impact between ETM 7.0 and 7.0.1 is minimal. * When we compared the performance of a much smaller component (size of 500 thousand versioned artifacts) versus the extremely large component (size of 10 million versioned artifacts): * The result shows the average response time for all pages have increased as the component size increases * 28 out of 128, or 22.8%, of the pages under test have an increase of at least 1 second, with the component size grows 20 times bigger; 9 pages have increased more than 3 seconds * The larger counts in an artifact view, especially in baselines, the more impact is by the flat growth of their component size, which can be seen from the list below showing top pages that are mostly sensitive to the component size increase: <p><img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/topPages_10Mvs500Kcmpnnt_in15Mrepo_table.png" /></p> * The server resource utilizations are the same or slightly increased. Automatic memory management (AMM) is enabled on Oracle server. For more details about the system resource including memory statistics of the Oracle database, please see section [[#Resource_utilization][Resource utilization]] and [[#Test_load_based_tunings][performance tunings]] of the test systems. ---++ Topology, network, and data volume ---+++Data volume and shape * ETM Database size on disk ~ 775 GB ---++++Versioned artifacts distribution by components The artifacts were created by the in-house datagen utility. For the large scale repository under test in this report, we created total of <i>15 million artifacts</i> that are spread out across one extremely large and ten smaller components, and for each of the component size the artifact counts are distributed as below: %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="versioned artifact counts by component size: " tableborder="2" tableframe="border" tablerules="none"}% |Counts|Extra large component |Smaller component| |test plans|1,681|50| |test cases|1,000,207|30,000| |test scripts|1,000,209|30,000| |test suites|100,020|3,000| |test case execution records|4,000,800|120,000| |test case results|4,000,921|360,000| |test suite results|500,100|15,000| |test execution schedules|47,467|1,424| |test phases and test environments|31,636|800| |build definitions and build records|33,533|1,006| |total|10,716,574|561,280| The artifact maintains the following associations: * each test plan is linked to 600 unique test cases, and 600 unique test scripts * each test suite contains 15 test cases in smaller -sized component, and 500 test cases in the extra large component, respectively ---++++Streams and baselines The initial component, which is the target for the automation test load, contains two streams and two baselines. The smaller component under test has one initial stream and one baseline created. ---++++Miscellaneous artifacts in the applications There are 5,000 work items (defects) in the CCM project area. ---+++ Topology and network The topology under test is based on Distributed / Linux / Database Server: <img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/ServerOverview.png" alt="ServerOverview.png" /> The specifications of machines under test are listed in the table below. Server tuning details listed in *[[#Appendix_Key_configuration_param][Appendix - Key configuration parameters]]* <table class="gray-table"> <tbody> <tr> <th valign="top">Function</th> <th valign="top">Number of Machines</th> <th valign="top">Machine Type</th> <th valign="top">CPU / Machine</th> <th valign="top">Total # of CPU vCores/Machine</th> <th valign="top">Memory/Machine</th> <th valign="top">Disk</th> <th valign="top">Disk capacity</th> <th valign="top">Network interface</th> <th valign="top">OS and Version</th> </tr> <tr> <td>Proxy Server (IBM HTTP Server and !WebSphere Plugin)</td> <td>1</td> <td>IBM System x3250 M4</td> <td>1 x Intel Xeon E3-1240 3.4GHz (quad-core)</td> <td>8</td> <td>16GB</td> <td>RAID 1 -- SAS Disk x 2</td> <td>299GB</td> <td>Gigabit Ethernet</td> <td>Red Hat Enterprise Linux Server release 6.5</td> </tr> <tr> <td>JTS Server</td> <td>1</td> <td>IBM System x3550 M4</td> <td>2 x Intel Xeon E5-2640 2.5GHz (six-core)</td> <td>24</td> <td>32GB</td> <td>RAID 5 -- SAS Disk x 2</td> <td>897GB</td> <td>Gigabit Ethernet</td> <td>Red Hat Enterprise Linux Server release 6.5</td> </tr> <tr> <td>ETM Server</td> <td>1</td> <td>IBM System x3550 M4</td> <td>2 x Intel Xeon E5-2640 2.5GHz (six-core)</td> <td>24</td> <td>32GB</td> <td>RAID 5 -- SAS Disk x 2</td> <td>897GB</td> <td>Gigabit Ethernet</td> <td>Red Hat Enterprise Linux Server release 6.5</td> </tr> <tr> <td>Database Server</td> <td>1</td> <td>IBM System x3650 M4</td> <td>2 x Intel Xeon E5-2640 2.5GHz (six-core)</td> <td>40</td> <td>64GB</td> <td>RAID 10 279GB SAS Disk x 16</td> <td>4TB</td> <td>Gigabit Ethernet</td> <td>Red Hat Enterprise Linux Server 7 (Maipo) </td> </tr> <tr> <td>RPT workbench</td> <td>1</td> <td>IBM System x3550 M4</td> <td>2 x Intel Xeon E5-2640 2.5GHz (six-core)</td> <td>24</td> <td>32GB</td> <td>RAID 5 -- SAS Disk x 2</td> <td>897GB</td> <td>Gigabit Ethernet</td> <td>Red Hat Enterprise Linux Server release 6.4</td> </tr> <tr> <td>RPT Agents</td> <td>6</td> <td>VM image</td> <td>4 x Intel Xeon X5650 CPU (1-Core 2.67GHz)</td> <td>1</td> <td>2GB</td> <td>N/A</td> <td>30GB</td> <td>Gigabit Ethernet</td> <td>Red Hat Enterprise Linux Server release 6.5</td> </tr> <tr> <td>Network switches</td> <td>N/A</td> <td>Cisco 2960G-24TC-L</td> <td>N/A</td> <td>N/A</td> <td>N/A</td> <td>N/A</td> <td>N/A</td> <td>Gigabit Ethernet</td> <td>24 Ethernet 10/100/1000 ports</td> </tr> </tbody> </table> N/A: Not applicable. vCores = Cores with hyperthreading ---+++Network connectivity All server machines and test clients are located on the same subnet. The LAN has 1000 Mbps of maximum bandwidth and less than 0.3 ms latency in ping. ---++ Performance load 1000 virtual users are executing test scenarios against ETM servers concurrently for one hour, generating a consistent throughput, a.k.a. page hit rate, of <b> 8.2 pages per second</b>, as shown in the picture below. During the concurrency load, we evaluated page loading for more than 100 individual pages by collecting their page response times (the smaller page response time, the better performance) and server statistics metric. <img src="https://jazz.net/wiki/pub/Deployment/CollaborativeLifecycleManagementPerformanceScalabilityReportRQM603Release/userload.png" /> Rational Performance Tester(RPT) is used to simulate this load created using the web client. Each virtual user completes a random use case from a set of available use cases. Each use case is implemented as a RPT script; and these RPT scripts are organized by pages representing user actions. The performance work load is role based as each of the areas defined under sequence of actions which are separated into individual user groups within an RPT schedule, see [[#User_roles_test_cases_and_worklo][User roles, test cases and workload characterization]] for detailed description for each test scenario. ---+++User roles, test cases and workload characterization *a. User Roles* #UserRoles <table class="gray-table"> <tbody> <tr> <th align="center" height="40"><b>Configuration</b></th> <th align="center" height="40"><b>Use role</b></th> <th align="center" height="40"><b>% of Total</b></th> <th align="center" height="40"><b>Related Actions</b></th> </tr> <tr> <td><b>Stream</b></td> <td align="center"></td> <td align="center"><b>98</b></td> <td></td> </tr> <tr> <td align="center"></td> <td>QE Manager</td> <td align="center">8</td> <td>Test plan create, Browse test plan and test case, Browse test script, Simple test plan copy, Defect search, View dashboard</td> </tr> <tr> <td align="center"></td> <td>Test Lead</td> <td align="center">19</td> <td>Edit Test Environments, Edit test plan, Create test case, Bulk edit of test cases, Full text search, Browse test script, Test Execution, Defect search</td> </tr> <tr> <td align="center"></td> <td>Tester</td> <td align="center">66</td> <td>Defect create, Defect modify, Defect search, Edit test case, Create test script, Edit test script, Test Execution, Browse test execution record</td> </tr> <tr> <td align="center"></td> <td>Dashboard Viewer</td> <td align="center">5</td> <td>View dashboard(with login and logout)</td> </tr> <tr> <td><b>Baseline</b></td> <td></td> <td align="center"><b>2</b></td> <td></td> </tr> <tr> <td align="center"></td> <td>Baseline Viewer</td> <td align="center">2</td> <td>Browse Test Case or Test Plan or Test Case Execution Record in baseline</td> </tr> </tbody> </table> *b. Test Cases* #TestCases <table class="gray-table"> <tbody> <tr> <th align="center" height="40"><b>Use Role</b></th> <th align="left" height="40"><b>Percentage of the user role</b></th> <th align="left" height="40"><b>Sequence of Operations</b></th> </tr> <tr> <td rowspan=6>QE Manager</td> <td align="center">1</td> <td>[[#TestPlanCreate][Test plan create]]:user creates test plan, then adds description, business objectives, test objectives, 2 test schedules, test estimate quality objectives and entry and exit criteria.</td> </tr> <tr> <td align="center">26</td> <td>[[#BrowseTestPlansAndTestCases][Browse test plans and test cases]]: user browses assets by: View Test Plans, then configure View Builder for name search; open test plan found, review various sections, then close. Search for test case by name, opens test case found, review various sections, then close.</td> </tr> <tr> <td align="center">26</td> <td>[[#BrowseTestScript][Browse test script]]: user search for test script by name, open it, reviews it, then closes.</td> </tr> <tr> <td align="center">1</td> <td>[[#SimpleTestPlanCopy][Simple test plan copy]]: user search test plan by name, then select one, then make a copy.</td> </tr> <tr> <td align="center">23</td> <td>[[#DefectSearch][Defect search]]: user searches for specific defect by number, user reviews the defect (pause), then closes.</td> </tr> <tr> <td align="center">20</td> <td>[[#ViewDashboardWithLogin][Refresh Dashboard]]: user views dashboard</td> </tr> <tr> <td rowspan=8>Test Lead</td> <td align="center">8</td> <td>[[#EditTestEnvironment][Edit Test Environment]]: user lists all test environments, and then selects one of the environments and modifies it.</td> </tr> <tr> <td align="center">15</td> <td>[[#EditTestPlan][Edit test plan]]: list all test plans; from query result, open a test plan for editing, add a test case to the test plan, a few other sections of the test plan are edited and then the test plan is saved.</td> </tr> <tr> <td align="center">4</td> <td>[[#TestCaseCreate][Create test case]]: user create test case by: opening the Create Test Case page, enters data for a new test case, and then saves the test case.</td> </tr> <tr> <td align="center">1</td> <td>[[#BulkEditofTestCases][Bulk edit of test cases]]: user searches for test cases with root name and edits all found with owner change.</td> </tr> <tr> <td align="center">0</td> <td>[[#FullTextSearch][Full text search]]: user does a full text search of all assets in repository using root name, then opens one of found items.</td> </tr> <tr> <td align="center">32</td> <td>[[#BrowseTestScript][Browse test script]]: user search for test script by name, open it, reviews it, then closes.</td> </tr> <tr> <td align="center">26</td> <td>[[#TestExecutionFor4Steps][Test Execution]]: selects View Test Execution Records, by name, starts execution, enters pass/fail verdict, reviews results, sets points then saves. </td> </tr> <tr> <td align="center">11</td> <td>[[#DefectSearch][Defect search]]: user searches for specific defect by number, user reviews the defect (pause), then closes.</td> </tr> <tr> <td rowspan=8>Tester</td> <td align="center">8</td> <td>[[#CreateDefect][Defect create]]: user creates defect by: opening the Create Defect page, enters data for a new defect, and then saves the defect. </td> </tr> <tr> <td align="center">5</td> <td>[[#DefectModify][Defect modify]]: user searches for specific defect by number, modifies it then saves it.</td> </tr> <tr> <td align="center">14</td> <td>[[#DefectSearch][Defect search]]: user searches for specific defect by number, user reviews the defect (pause), then closes.</td> </tr> <tr> <td align="center">6</td> <td>[[#EditTestCase][Edit test case]]: user searches Test Case by name, the test case is then opened in the editor, then a test script is added to the test case (user clicks next a few times (server size paging feature) before selecting test script), The test case is then saved.</td> </tr> <tr> <td align="center">4</td> <td>[[#TestScriptCreate][Create test script]]: user creates test case by: selecting Create Test Script page, enters data for a new test script, and then saves the test script. </td> </tr> <tr> <td align="center">8</td> <td>[[#EditTestScript][Edit test script]]: user selects Test Script by name. test script then opened for editing, modified and then saved.</td> </tr> <tr> <td align="center">42</td> <td>[[#TestExecutionFor4Steps][Test Execution]]: selects View Test Execution Records, by name, starts execution, enters pass/fail verdict, reviews results, sets points then saves. </td> </tr> <tr> <td align="center">7</td> <td>[[#TestExecutionRecordBrowsing][Browse test execution record]]: user browses TERs by: name, then selects the TER and opens the most recent results.</td> </tr> <tr> <td>Dashboard Viewer</td> <td align="center">100</td> <td>[[#ViewDashboardWithLogin][View dashboard(with login and logout)]]: user logs in, views dashboard, then logs out. This user provides some login/logout behavior to the workload </td> </tr> <tr> <td rowspan=6>Baseline Viewer</td> <td align="center">34</td> <td>[[#BrowseTestScriptInBaseline][Browse test script in baseline]]: user selects browse test scripts, and opens a test script from the list </td> </tr> <tr> <td align="center">33</td> <td>[[#TERBrowsingInBaseline][Browse test case execution record in baseline]]: user selects browse test case execution records, opens one of them from the list </td> </tr> <tr> <td align="center">33</td> <td>[[#BrowseTPsTCsInBaseline][Browse test plans and test cases in baseline]]: user browses assets by: View Test Plans, then configure View Builder for name search; open test plan found, review various sections, then close. Search for test case by name, opens test case found, review various sections, then close.</td> </tr> </tbody> </table> ---+++Response time comparison The page performance is measured as mean value (or average) of its response time in the result data. For the majority of the pages under tests, there is little variation between runs, and the mean values are close to median in the sample for the load. ---++ Detailed performance results #DetailedResults ---+++Average page response time breakdown *NOTE* For all charts, each column represents the time for page loading (aka page response time) in milliseconds, for which value smaller is better. #BrowseTestPlansAndTestCases Browse Test Plans & Test Cases %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Browse Test Plans & Test Cases" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/browseTCs.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/browseTCs_byCompSize.png" />| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/browseTPs.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/browseTPs_byCompSize.png" />| |<p style="color:black;font-size:10px;">* Note: this page can be variable; however the 90th percentile of this page is under 1 second in both cases. </p>| [[#TestCases][Back to Test Cases & workload characterization]] #BrowseTestScript Browse Test Scripts %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Browse Test Scripts" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/browseTestScripts.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/browseTestScripts_byCompSize.png" />| [[#TestCases][Back to Test Cases & workload characterization]] #TestExecutionRecordBrowsing Test Execution Record Browsing %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Browse Test Execution Record" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/browseTERs.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/browseTERs_byCompSize.png" />| [[#TestCases][Back to Test Cases & workload characterization]] #CreateDefect Create Defect %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Create Defect" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/createDefect.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/createDefect_byCompSize.png" />| [[#TestCases][Back to Test Cases & workload characterization]] #TestPlanCreate Create Test Plan %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Create Test Plan" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/createTestPlan.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/createTestPlan_byCompSize.png" />| <p> [[#TestCases][Back to Test Cases & workload characterization]] #TestCaseCreate Create Test Case %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Create Test Case" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/createTestCase.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/createTestCase_byCompSize.png" />| [[#TestCases][Back to Test Cases & workload characterization]] #TestScriptCreate Create Test Script %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Create Test Script" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/createTestScript.png"/>|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/createTestScript_byCompSize.png"/>| [[#TestCases][Back to Test Cases & workload characterization]] #EditTestCase Edit Test Case %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Edit Test Case" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/editTestCase.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/editTestCase_byCompSize.png" />| <p> [[#TestCases][Back to Test Cases & workload characterization]] #EditTestEnvironment Edit Test Environment %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Edit Test Environment" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/editTestEnv.png"/>|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/editTestEnv_byCompSize.png"/>| [[#TestCases][Back to Test Cases & workload characterization]] #EditTestPlan Edit Test Plan %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Edit Test Plan" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/editTestPlan.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/editTestPlan_byCompSize.png" />| [[#TestCases][Back to Test Cases & workload characterization]] #EditTestScript Edit Test Script %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Edit Test Script" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/editTestScript.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/editTestScript_byCompSize.png" />| [[#TestCases][Back to Test Cases & workload characterization]] #TestExecutionFor4Steps Test Execution For 4 Steps %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Test Execution For 4 Steps" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/executeTER.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/executeTER_byCompSize.png" />| [[#TestCases][Back to Test Cases & workload characterization]] #SimpleTestPlanCopy Simple Test Plan Copy %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Simple Test Plan Copy" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/simpleCopy.png"/>|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/simpleCopy_byCompSize.png"/>| [[#TestCases][Back to Test Cases & workload characterization]] #BulkEditofTestCases Bulk Edit of Test Cases %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Bulk Edit of Test Cases" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/bulkEdit.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/bulkEdit_byCompSize.png" />| [[#TestCases][Back to Test Cases & workload characterization]] #ViewDashboardWithLogin View Dashboard %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="View Dashboard" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/viewDashboard.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/viewDashboard_byCompSize.png" />| <p> [[#TestCases][Back to Test Cases & workload characterization]] #BrowseTestScriptInBaseline Browse Test Script In Baseline %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Browse Test Script In Baseline" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/browseTestScripts_inBaseline.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/browseTestScripts_inBaseline_byCompSize.png" />| <p> [[#TestCases][Back to Test Cases & workload characterization]] #TERBrowsingInBaseline Browse Test Case Execution Record In Baseline %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Browse Test Case Execution Record In Baseline" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/browseTERs_inBaseline.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/browseTERs_inBaseline_byCompSize.png" />| <p> [[#TestCases][Back to Test Cases & workload characterization]] #BrowseTPsTCsInBaseline Browse Test Plans And Test Cases In Baseline %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Browse Test Plans And Test Cases In Baseline" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/browseTPsTCs_inBaseline.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/browseTPsTCs_inBaseline_byCompSize.png" />| <p> [[#TestCases][Back to Test Cases & workload characterization]] #DefectSearch Defect Search %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Defect Search" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/searchDefect.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/searchDefect_byCompSize.png" />| [[#TestCases][Back to Test Cases & workload characterization]] #DefectModify Defect Modify %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Defect Modify" tableborder="2" tableframe="border" tablerules="none"}% | repository size increased by 50%|component size increased by 2,000%| |<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/modifyDefect.png" />|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/modifyDefect_byCompSize.png" />| [[#TestCases][Back to Test Cases & workload characterization]] ---+++RPT script executions RPT test scenario breakdown: average page executions during the one-hour staging period <font size=1> %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="" tableborder="2" tableframe="border" tablerules="none" }% |<img src="https://jazz.net/wiki/pub/Deployment/CollaborativeLifecycleManagementPerformanceAndScalabilityReportRationalQualityManager6061Ifx001/scriptExecutionCounts.png" />| *Test scenario* | *Average execution counts per load test*| |^|[[#TestExecutionFor4Steps][Execute the TER (Test Execution For 4 Steps)]]|1,478| |^|[[#TestExecutionRecordBrowsing][Browse TER]]|864| |^|[[#DefectSearch][Search Defect]]|650| |^|[[#ViewDashboardWithLogin][View Dashboard]]|508| |^|[[#BrowseTestScript][Browse Test Script]]|460| |^|[[#EditTestScript][Edit Test Script]]|245| |^|[[#CreateDefect][Create Defect]]|222| |^|[[#EditTestCase][Edit Test Case]]|170| |^|[[#EditTestPlan][Edit Test Plan]]|163| |^|[[#DefectModify][Modify Defect]]|150| |^|[[#TestScriptCreate][Create Test Script]]|119| |^|[[#BrowseTestPlansAndTestCases][Browse Test Plans And Test Cases]]|110| |^|[[#ViewDashboardWithLogin][Refresh Dashboard]]|98| |^|[[#EditTestEnvironment][Edit Test Environment]]|91| |^|[[#TestCaseCreate][Create Test Case]]|64| |^|[[#BrowseTestScriptInBaseline][Browse Test Script In Baseline]]|23| |^|[[#TERBrowsingInBaseline][Browse TER In Baseline]]|23| |^|[[#BrowseTPsTCsInBaseline][Browse Test Plans And Test Cases In Baseline]]|17| |^|[[#FullTextSearch][Full Text Search and Open Test Suite]]|0| |^|[[#BulkEditofTestCases][Bulk Edit of Test Cases]]|17| |^|[[#TestPlanCreate][Create Test Plan]]|2| |^|[[#SimpleTestPlanCopy][Simple Test Plan Copy]]|2| </font> ---+++RPT network transfer comparison <img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/serverThroughputs.png" /> ---+++Resource utilization %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="Database Server resource utilization - ETM7.0.1 15 million repository - 1000users performance load large scale: " tableborder="2" tableframe="border" tablerules="none"}% ||extra large component vs smaller component || |CPU utilization|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/1serverutil_CPU_oracle.png" />|Disk utilization|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/1serverutil_DiskBusyness_oracle.png" />| |Memory|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/1serverutil_Memory_oracle.png" />|Disk transfer|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/1serverutil_DiskXfer_oracle.png" />| %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="WebSphere Liberty Server (ETM) resource utilization - ETM7.0.1 15 million repository - 1000users performance load large scale: " tableborder="2" tableframe="border" tablerules="none"}% ||extra large component vs smaller component|| |CPU utilization|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/2serverutil_CPU_QMAppSvr.png" />|Disk utilization|<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/2serverutil_DiskBusyness_QMAppSvr.png" />| ---+++Garbage collection - JVM GC Chart For JVM parameter please refer to [[#Appendix_Key_configuration_param][Appendix - Key configuration parameters]] Verbose garbage collection is enabled to log the GC activities. These GC logs showed very little variation between runs. There is also no discernible difference between versions. Below is one example of the output from the GC log for each application. %TABLE{ sort="off" headerbg="#3399FF" cellpadding="2" cellspacing="2" dataalign="center" caption="WebSphere Liberty Server JVM Garbage Collections" tableborder="2" tableframe="border" tablerules="none"}% ||<img src="https://jazz.net/wiki/pub/Deployment/ELMLargeScaleAndPerformanceReportRelease701/gcChart_qm.png" />| ---++Test load based tunings The numbers represented in the report are based on some levels of optimization techniques. ---+++Apply additional indexes based on the test load As seen from the [[#RPT_script_executions][load execution breakdown]], the ETM 7.0.1 performance workload contains Test Case Execution-intensive usage scenarios by design, where more than 40% of the virtual users are either browsing or searching a Test Case Execution Record, or executing it in a stream. This has resulted in the default <i>Browse Test Case Execution Record (Open TER view)</i> page being consistently loaded with high frequency. Another usage pattern for this workload is to apply the <i>Name</i> filter for all other types of artifacts across different test scenarios, which contribute to 20% of the total page loadings. The following chart shows an example of applying the <i>Name</i> column filter and a typical sort, implemented in the performance load: <img src="https://jazz.net/wiki/pub/Deployment/CollaborativeLifecycleManagementPerformanceAndScalabilityReportRationalQualityManager6061Ifx001/testscenario_browseTERsearchTER.png" alt="searching TER using a keyword in the name filter in result table" />. These pages could be the most expensive ones in our performance test, taking more than 10 seconds to load for instance. When we looked at the queries from within the database, we determined to use additional indexes to help improve both of their response times as well as the memory usage. The indexes were created on the ETM database schema as below <verbatim> --function-based indexes for using name column for filtering and searching CREATE INDEX "QM_USER"."NAMECOL_UPPER_TER" ON "QM_USER"."EXECUTION_EXECUTION_RECORD" (UPPER("NAME")); CREATE INDEX "QM_USER"."NAMECOL_UPPER_TCER" ON "QM_USER"."EXECUTION_EXECUTION_TASK_RESLT" (UPPER("NAME")); CREATE INDEX "QM_USER"."NAMECOL_UPPER_TS" on "QM_USER"."PLANNING_EXECUTION_SCRIPT" (UPPER(NAME)); CREATE INDEX "QM_USER"."NAMECOL_UPPER_CATG" on "QM_USER"."PLANNING_CATEGORY" (UPPER(NAME)); CREATE INDEX "QM_USER"."NAMECOL_UPPER_TC" on "QM_USER"."PLANNING_TEST_CASE" (UPPER(NAME)); CREATE INDEX "QM_USER"."NAMECOL_UPPER_TSU" on "QM_USER"."PLANNING_TEST_SUITE" (UPPER(NAME)); CREATE INDEX "QM_USER"."NAMECOL_UPPER_CTRR" on "QM_USER"."REPOSITORY_CONTRIBUTOR_RECORD" (UPPER(NAME)); CREATE INDEX "QM_USER"."USERID_UPPER_CTRR" on "QM_USER"."REPOSITORY_CONTRBTR_RCRD_SR_DS" (UPPER(USER_ID)); --composite indexes to improve large views such as execution record, and test results CREATE INDEX "QM_USER"."TER_JZ_ARC_MOD_PRJ_ITM_ST" ON "QM_USER"."EXECUTION_EXECUTION_RECORD" ("JZ_DISCRIMINATOR", "ARCHIVED", "MODIFIED", "PROJECT_AREA_ITEM_ID", "ITEM_ID", "STATE_ID"); CREATE INDEX "QM_USER"."TCER_JZ_ARC_ET_PRJ_ITM_ST" ON "QM_USER"."EXECUTION_EXECUTION_TASK_RESLT" ("JZ_DISCRIMINATOR", "ARCHIVED", "END_TIME", "PROJECT_AREA_ITEM_ID", "ITEM_ID", "STATE_ID"); CREATE INDEX "QM_USER"."JZ_ARC_PRJ_SNPS_ITM_ST" ON "QM_USER"."PLANNING_VERSIONABLE_ITEM" ("JZ_DISCRIMINATOR", "ARCHIVED", "PROJECT_AREA_ITEM_ID", "SNAPSHOT", "ITEM_ID", "STATE_ID"); --composite index for process CREATE INDEX "QM_USER"."PRJITMID_ITMID" ON "QM_USER"."PROCESS_PROCESS_AREA" ("PROJECT_AREA_ITEM_ID", "ITEM_ID"); </verbatim> ---+++Apply SQL profile We created function-based indexes for the queries for filtering/searching, as shown above. However they may not be directly recognized yet applied by the underlying queries. To make them immediately effective, we used SQL profile suggested by the Oracle SQL Tuning Advisor. The process to apply the SQL profiles is: * emulate the searching and filtering scenarios, via web browser or automation * collect theses queries (this can be done either from tracking the hotspot queries from the Oracle Performance Hub, or found in the AWR report) * create tuning tasks, and execute tuning tasks * review tuning suggestion, make sure the function-based indexes are being used by the EXPLAIN plan with which performance is improved * accept tuning task that applies the SQL profile In ETM 7.0.1, we found that the option to allow function-based indexes through SQL profiling may disappear due to an index "RPSTR_VRSNSTDBDSCRNTTMCRTDCNFG" presented under the ETM user schema. After dropping this index, the EXPLAIN PLAN from SQL profile will become desired. Based on the performance payload test and extended longer loads, dropping this particular index in the ETM user schema does not seem to introduce negative performance impact to the application. ---+++Oracle tuning parameters The recommended Oracle tunings for ELM is documented in the [[https://jazz.net/wiki/bin/view/Deployment/RequirementsManagement70Performance#Oracle_tuning][DNG performance guide]]. The test results in this report adopted the majority of the settings or concepts, except for leaving <b>filesystemio_options</b> and <b>parallel_degree_policy</b> default values. While they can further improve some of the page response times to a great deal, we found they could also made the load variable and response times unstable if the disk busyness% on the Oracle server is already high. We recommend to tune them accordingly when you have extra disk capacity. The tuning parameters for ETM7.0.1 large scale test are: <verbatim> ####enable automatic memory management (AMM) #original memory_max_target and memory_target are both 0; #original sga_target=28800M and sga_max_size=40G; #may need to increase the memory settings on the system first, run as root: mount -t tmpfs shmfs -o size=56g /dev/shm alter system set memory_max_target=52000M scope=spfile; alter system set memory_target=52000M scope=spfile; alter system set sga_max_size=0 scope=spfile; alter system set sga_target=0 scope=spfile; alter system set pga_aggregate_limit=0 scope=spfile; alter system set pga_aggregate_target=0 scope=spfile; #rebounce db. note that in the test we also secured the sizes of sga_max_size and db_cache_size to match the high water mark observed prior to tuning alter system set sga_max_size=40G scope=spfile; alter system set db_cache_size=24G scope=spfile; #rebounce db ####disable adaptive plan alter system set optimizer_adaptive_plans = false #default true #_optimizer_use_feedback= false #not available in 12c ####these settings can help enable auto parallelism in SQL execution. however for high payload, only turn these on when you have enough disk capacity #alter system set parallel_degree_policy=auto scope=spfile; #default manual #alter system set filesystemio_options=setall scope=spfile; #default none #rebounce db </verbatim> ---++Appendix - Key configuration parameters <table class="gray-table"> <tbody> <tr> <th align="left" width="200"><strong>Product</strong><br></th> <th align="left" width="100"><strong>Version</strong></th> <th align="left" width="400"><strong>Highlights for configurations under test</strong></th> </tr> <tr> <td style="vertical-align: top;">IBM HTTP Server for !WebSphere Application Server</td> <td style="vertical-align: top;">8.5.5.12</td> <td style="vertical-align: top;">IBM HTTP Server functions as a reverse proxy server implemented via Web server plug-in for !WebSphere Application Server. Configuration details can be found from the [[http://pic.dhe.ibm.com/infocenter/clmhelp/v4r0/index.jsp?topic=%2Fcom.ibm.jazz.install.doc%2Ftopics%2Ft_config_reverse_proxy_ihs.html][CLM infocenter]]. *HTTP server (httpd.conf)*: * !MaxClients: increase value for high-volume loads [[http://publib.boulder.ibm.com/httpserv/ihsdiag/ihs_performance.html][(adjust value based on user load)]] * !ThreadsPerChild = 25 *OS Configuration*: * max user processes = unlimited </td> </tr> <tr> <td style="vertical-align: top;">Embedded WebSphere Liberty Server</td> <td style="vertical-align: top;">20.0.0.3 </td> <td style="vertical-align: top;"><strong>JVM settings:</strong> * GC policy and arguments, max and init heap sizes: <verbatim> -Xgcpolicy:gencon -Xmx16g -Xms16g -Xmn4g -Xcompressedrefs -Xgc:preferredHeapBase=0x100000000 -verbose:gc -Xverbosegclog:gc.log -XX:MaxDirectMemorySize=1G </verbatim> *SDK version:* * java.version = 1.8.0_191; java.runtime = Java(TM) SE Runtime Environment (8.0.5.25 - pxa6480sr5fp25-20181030_01(SR5 FP25)) *Thread pools:* * Maximum !WebContainer = Minimum !WebContainer = 500 *OS Configuration:* System wide resources for the app server process owner: * max user processes = unlimited * open files = 65536 </td> </tr> <tr> <td>Oracle</td> <td>Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production</td> <td> </td> </tr> <tr> <td>LDAP server</td> <td> </td> <td> </td> </tr> <tr> <td>License server</td> <td> </td> <td>N/A</td> </tr> <tr> <td>RPT workbench</td> <td>8.3.0.3</td> <td>Defaults</td> </tr> <tr> <td>RPT agents</td> <td>8.3.0.3</td> <td>Defaults</td> </tr> <tr> <td>Network</td> <td> </td> <td>Shared subnet within test lab</td> </tr> </tbody> </table> -------------------- ---++++!! About the authors: Main.HongyanHuo is a performance engineer focusing on the performance test & analysis of products in the Collaborative Lifecycle Management family. <br> ---++++!! Related topics: [[CollaborativeLifecycleManagementPerformanceReportRQM60Release_v2][Collaborative Lifecycle Management performance report: Rational Quality Manager 6.0 release]], [[PerformanceDatasheetsAndSizingGuidelines][Performance datasheets]] -------------------- ---+++++!! Questions and comments: * What other performance information would you like to see here? * Do you have performance scenarios to share? * Do you have scenarios that are not addressed in documentation? * Where are you having problems in performance? %COMMENT{type="below" target="PerformanceDatasheetReaderComments" button="Submit"}% %INCLUDE{"PerformanceDatasheetReaderComments"}% <sticky></div></sticky>
Edit
|
Attach
|
P
rintable
|
V
iew topic
|
Backlinks:
We
b
,
A
l
l Webs
|
H
istory
:
r19
|
r17
<
r16
<
r15
<
r14
|
More topic actions...
Deployment
Deployment web
Planning and design
Installing and upgrading
Migrating and evolving
Integrating
Administering
Monitoring
Troubleshooting
Community information and contribution guidelines
Create new topic
Topic list
Search
Advanced search
Notify
RSS
Atom
Changes
Statistics
Web preferences
NOTE: Please use the Sandbox web for testing
Status icon key:
To do
Under construction
New
Updated
Constant change
None - stable page
Smaller versions of status icons for inline text:
Copyright © by IBM and non-IBM contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our
Terms of Use.
Please read the following
disclaimer
.
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more
here
.