(r16) CollaborativeLifecycleManagementPerformanceReportRDNG60Release < Deployment

Engineering Lifecycle Management Wiki - Deployment
Community information and contribution guidelines
~~EditAttach~~Printable
TWiki > Deployment Web > RwattsSandbox > DeploymentPlanningAndDesign > PerformanceDatasheetsAndSizingGuidelines > CollaborativeLifecycleManagementPerformanceReportRDNG60Release
Revision 16 - 2015-05-19 - 02:46:25 - Main.gsven
<div id="header-title" style="padding: 10px 15px; border-width:1px; border-style:solid; border-color:#FFD28C; background-image: url(<nop>https://jazz.net/wiki/pub/Deployment/WebPreferences/TLASE.jpg); background-size: cover; font-size:120%">
---+!! <img src="https://jazz.net/wiki/pub/Deployment/WebPreferences/new.png" alt="new.png" width="50" height="50" align="right"> Collaborative Lifecycle Management performance report: RDNG 6.0</br>
%DKGRAY% Authors: Main.GustafSvensson, Main.VaughnRokosz <br>
Date: May 22, 2015 </br>
Build basis: Rational Doors Next Generation 6.0

%ENDCOLOR%</div></sticky>

<!-- Page contents top of page on right hand side in box -->
<sticky><div style="float:right; border-width:1px; border-style:solid; border-color:#DFDFDF; background-color:#F6F6F6; margin:0 0 15px 15px; padding: 0 15px 0 15px;">
%TOC{title="Page contents"}%
</div></sticky>

<sticky><div style="margin:15px;"></sticky>

---++ Introduction

This report compares the performance of IBM Rational DOORS Next Generation version 6.0 to the previous 5.0.2 release.    The goal of the test was to verify that the performance of 6.0 operations were the same or better than their 5.0.2 equivalents.

The test methodology involves these steps:
   * Run a standard 250 user workload against version 5.0.2 for 1.5 hours (repeat 3 times)
   * Run a standard 250 user workload against version 6.0 for 1.5 hours (repeat 3 times)
   * Compare the 6 runs.    

The 6.0 tests do not include any of the new 6.0 features (such as the configuration management capabilities), which allows the performance workload and data repositories to be the same for 5.0.2 and 6.0.

---+++ Findings

The 6.0 release performs the same or better than the 5.0.2 release, when using the 6.0 release with the new configuration management features disabled.

<expand>
---+++!! Disclaimer

%INCLUDE{"PerformanceDatasheetDisclaimerEndToEnd"}%

---++ Test details

---+++ Topology

The topology under test is based on [[RecommendedCLMDeploymentTopologies5#CLM_E1_Enterprise_Distributed_Li][Standard Topology (E1) Enterprise - Distributed / Linux / DB2.]]

<img src="%ATTACHURLPATH%/ServerTopology.png" alt="Server Overview" width="60%" height="60%" />

The specifications of machines under test are listed in the table below. Server tuning details listed in *[[#AppendixA][Appendix A]]*

<table class="gray-table">
	<tbody>
		<tr>
			<th valign="top">Function</th>
			<th valign="top">Number of Machines</th>
			<th valign="top">Machine Type</th>
			<th valign="top">CPU / Machine</th>
			<th valign="top">Total # of CPU Cores/Machine</th>
			<th valign="top">Memory/Machine</th>
			<th valign="top">Disk</th>
			<th valign="top">Disk capacity</th>
			<th valign="top">Network interface</th>
			<th valign="top">OS and Version</th>
		</tr>
		<tr>
			<td>Proxy Server (IBM HTTP Server and !WebSphere Plugin)</td>
			<td>1</td>
			<td>IBM System x3250 M4</td>
			<td>1 x Intel Xeon E3-1240 3.4GHz (quad-core)</td>
			<td>8</td>
			<td>16GB</td>
			<td>RAID 1 -- SAS Disk x 2</td>
			<td>279GB</td>
			<td>Gigabit Ethernet</td>
			<td>Red Hat Enterprise Linux Server release 6.3 (Santiago)</td>
		</tr>
		<tr>
			<td>JTS Server</td>
			<td>1</td>
			<td>IBM System x3550 M4</td>
			<td>2 x Intel Xeon E5-2640 2.5GHz (six-core)</td>
			<td>24</td>
			<td>32GB</td>
			<td>RAID 5 -- SAS Disk x 4</td>
			<td>279GB</td>
			<td>Gigabit Ethernet</td>
			<td>Red Hat Enterprise Linux Server release 6.3 (Santiago)</td>
		</tr>
		<tr>
			<td>RDNG Server</td>
			<td>1</td>
			<td>IBM System x3550 M4</td>
			<td>2 x Intel Xeon E5-2640 2.5GHz (six-core)</td>
			<td>24</td>
			<td>32GB</td>
			<td>RAID 5 -- SAS Disk x 4</td>
			<td>279GB</td>
			<td>Gigabit Ethernet</td>
			<td>Red Hat Enterprise Linux Server release 6.3 (Santiago)</td>
		</tr>
		<tr>
			<td>Database Server</td>
			<td>1</td>
			<td>IBM System x3650 M4</td>
			<td>2 x Intel Xeon E5-2640 2.5GHz (six-core)</td>
			<td>24</td>
			<td>64GB</td>
			<td>RAID 10 -- SAS Disk x 16</td>
			<td>279GB</td>
			<td>Gigabit Ethernet</td>
			<td>Red Hat Enterprise Linux Server release 6.3 (Santiago)</td>
		</tr>
		<tr>
			<td>RPT Workbench</td>
			<td>1</td>
			<td>VM image</td>
			<td>2 x Intel Xeon X7550 CPU (1-Core 2.0GHz 64-bit)</td>
			<td>2</td>
			<td>6GB</td>
			<td>SCSI</td>
			<td>80GB</td>
			<td>Gigabit Ethernet</td>
			<td>Microsoft Windows Server 2003 R2 Standard Edition SP2</td>
		</tr>
		<tr>
			<td>RPT Agent</td>
			<td>1</td>
			<td>xSeries 345</td>
			<td>4 x Intel Xeon X3480 CPU (1-Core 3.20GHz 32-bit)</td>
			<td>4</td>
			<td>3GB</td>
			<td>SCSI</td>
			<td>70GB</td>
			<td>Gigabit Ethernet</td>
			<td>Microsoft Windows Server 2003 Enterprise Edition SP2</td>
		</tr>
		<tr>
			<td>RPT Agent</td>
			<td>1</td>
			<td>xSeries 345</td>
			<td>4 x Intel Xeon X3480 CPU (1-Core 3.20GHz 32-bit)</td>
			<td>4</td>
			<td>3GB</td>
			<td>RAID 1 - SCSI Disk x 2</td>
			<td>70GB</td>
			<td>Gigabit Ethernet</td>
			<td>Microsoft Windows Server 2003 Enterprise Edition SP2</td>
		</tr>
		<tr>
			<td>RPT Agent</td>
			<td>1</td>
			<td>Lenovo 9196A49</td>
			<td>1 x Intel Xeon E6750 CPU (2-Core 2.66GHz 32-bit)</td>
			<td>2</td>
			<td>2GB</td>
			<td>SATA</td>
			<td>230GB</td>
			<td>Gigabit Ethernet</td>
			<td>Microsoft Windows Server 2003 Enterprise Edition SP2</td>
		</tr>
		<tr>
			<td>Network switches</td>
			<td>N/A</td>
			<td>Cisco 2960G-24TC-L</td>
			<td>N/A</td>
			<td>N/A</td>
			<td>N/A</td>
			<td>N/A</td>
			<td>N/A</td>
			<td>Gigabit Ethernet</td>
			<td>24 Ethernet 10/100/1000 ports</td>
		</tr>
	</tbody>
</table>



---+++Network connectivity

All server machines and test clients are located on the same subnet. The LAN has 1000 Mbps of maximum bandwidth and less than 0.3ms latency in ping.

---+++Data volume and shape

The artifacts were distributed between 6 projects for a total of 517,080 artifacts.

The repository contained the following data:

   * 312 modules
   * 510,000 module artifacts
   * 7,800 requirement artifacts
   * 715 folders
   * 150 collections
   * 0 reviews
   * 1,563,492 comments
   * 1800 public tags
   * 0 private tags
   * 1,428 terms
   * 1,824,174 links
   * 600 views


   * JTS index size on disk = 1.2 MB
   * RM index size on disk = 41 GB

Each project contained the following data:

   * 52 modules
   * 119 folders
   * 1,180 requirement artifacts
   * 85,000 module artifacts
   * 14 collections
   * 0 reviews
   * 260,582 comments
   * 300 public tags
   * 0 private tags
   * 238 terms
   * 304,029 links
   * 100 views

---++Methodology

Rational Performance Tester was used to simulate the workload created using the web client. Each user completed a random use case from a set of available use cases. A Rational Performance Tester script is created for each use case. The scripts are organized by pages and each page represents a user action.

Based on real customer use, the test scenario provides a ratio of 70% reads and 30% writes. The users completed use cases at a rate of 60 pages per
hour per user. Each performance test runs for 90 minutes after all of the users are activated in the system.

---+++Test cases and workload characterization
#TestCases
<table class="gray-table">
	<tbody>
		<tr>
			<th align="center">Use case</th>
			<th align="center">Description</th>
			<th align="center">% of Total Workload</th>
		</tr>
		<tr>
			<td>Login</td>
			<td>Connect to the server using server credentials.</td>
			<td align="center">0</td>
		</tr>
		<tr>
			<td>[[#CreateCollection][Create a collection]]</td>
			<td>Create collections with 20 artifacts.</td>
			<td align="center">2</td>
		</tr>
		<tr>
			<td>[[#QueryByIdAndString][Query by ID and string]]</td>
			<td>Within scope of a project search for a known artifact by using the ID and also by string (two separate searches)</td>
			<td align="center">9</td>
		</tr>
		<tr>
			<td>[[#ManageFolders][Manage folders]]</td>
			<td>Create a folder, move it to a new location, and then delete the folder.</td>
			<td align="center">2</td>
		</tr>
		<tr>
			<td>[[#CreateModArtifactComment][Add comments to a module artifact]]</td>
			<td>Open a module artifact and create a comment addressed to a person on the team.</td>
			<td align="center">19</td>
		</tr>
		<tr>
			<td>[[#OpenProjectDashboard][Open the project dashboard]]</td>
			<td>Open the project and dashboard for the first time.</td>
			<td align="center">5</td>
		</tr>
		<tr>
			<td>[[#CreateMedModArtifactE2E][Create a module artifact in a medium sized module]]</td>
			<td>Open module with 500 requirements. Create a module artifact, edit three attributes and then delete the artifact from the module.</td>
			<td align="center">6</td>
		</tr>
		<tr>
			<td>[[#CreateModArtifactE2E][Create a module artifact in a large module]]</td>
			<td>Open module with 1500 requirements.  Create a module artifact, edit three attributes and then delete the artifact from the project.</td>
			<td align="center">19</td>
		</tr>
		<tr>
			<td>[[#CopyPaste25ModArtifacts][Copy and paste module artifacts in a large module]]</td>
			<td>Open module with 1500 requirements.  Create 24 new requirements by copy and paste 24 existing requirements.</td>
			<td align="center">1</td>
		</tr>
		<tr>
			<td>[[#CreateArtifact][Create an artifact]]</td>
			<td>Create a requirement that has 50 enumerations. Open the artifact and add three enumeration attributes</td>
			<td align="center">3</td>
		</tr>
		<tr>
			<td>[[#HoverOpenCollection][Display the hover information for a collection]]</td>
			<td>Open Show Collections and hover over a collection.</td>
			<td align="center">2</td>
		</tr>
		<tr>
			<td>[[#HoverEditModArtifact][Hover over module artifact]]</td>
			<td>Open module.  Display hover information for one of the artifacts.  Edit and save the module</td>
			<td align="center">2</td>
		</tr>
		<tr>
			<td>[[#ScrollModule][Open a module and page down]]</td>
			<td>Open a module with 1500 artifacts.  Page down 20 pages, one page at a time.</td>
			<td align="center">19</td>
		</tr>
		<tr>
			<td>[[#SwitchModuleView][Change columns to display in a module]]</td>
			<td>Open a module with 1500 artifacts. Add a column to the display.  Filter the display by an attribute</td>
			<td align="center">14</td>
		</tr>
<!--	<tr>
			<td>[[#DisplayModuleHistory][Display module history]]</td>
			<td>Open module and display history and audit history.</td>
			<td align="center">1</td>
		</tr>
-->		
                <tr>
			<td>[[#Upload4MBFileInNewArtifact][Upload 4MB file in new artifact]]</td>
			<td>Upload 4MB file.</td>
			<td align="center">4</td>
		</tr>
	</tbody>
</table>

<table border="1" cellpadding="0" cellspacing="0" id="_396" style="margin-left:5px">
   <tbody>
      <tr>
         <th colspan="1" rowspan="1" style="width: 250px; vertical-align: center; ">
            <p dir="ltr" id="_397" style="margin-left: 5px; "><b>Use case</b></p>
         </th>
         <th colspan="1" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_398" style="margin-left: 5px; "><b>Workload</b></p>
         </th>
         <th colspan="1" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_399" style="margin-left: 5px; "><b>Number of times executed in 5.0.2</b></p>
         </th>
         <th colspan="1" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_400" style="margin-left: 5px; "><b>Number of times executed in 6.0</b></p>
         </th>
      </tr>
      <tr>
         <td colspan="1" id="_401" rowspan="1" style="width: 250px; vertical-align: center; ">
            <p dir="ltr" id="_402" style="margin-left: 5px; ">Copy and paste module artifacts in a large module</p>
         </td>
         <td colspan="1" id="_403" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_404" style="margin-left: 5px; ">1%</p>
         </td>
         <td colspan="1" id="_405" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_406" style="margin-left: 5px; ">40</p>
         </td>
         <td colspan="1" id="_500" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_501" style="margin-left: 5px; ">38</p>
         </td>
      </tr>
      <tr>
         <td colspan="1" id="_407" rowspan="1" style="width: 250px; vertical-align: center; ">
            <p dir="ltr" id="_408" style="margin-left: 5px; ">Create an artifact</p>
         </td>
         <td colspan="1" id="_409" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_410" style="margin-left: 5px; ">3%</p>
         </td>
         <td colspan="1" id="_411" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_412" style="margin-left: 5px; ">121</p>
         </td>
         <td colspan="1" id="_502" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_503" style="margin-left: 5px; ">125</p>
         </td>
      </tr>
      <tr>
         <td colspan="1" id="_413" rowspan="1" style="width: 250px; vertical-align: center; ">
            <p dir="ltr" id="_414" style="margin-left: 5px; ">Create a collection</p>
         </td>
         <td colspan="1" id="_415" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_416" style="margin-left: 5px; ">2%</p>
         </td>
         <td colspan="1" id="_417" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_418" style="margin-left: 5px; ">40</p>
         </td>
         <td colspan="1" id="_504" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_505" style="margin-left: 5px; ">41</p>
         </td>
      </tr>
      <tr>
         <td colspan="1" id="_419" rowspan="1" style="width: 250px; vertical-align: center; ">
            <p dir="ltr" id="_420" style="margin-left: 5px; ">Create a large module artifact end-to-end scenario</p>
         </td>
         <td colspan="1" id="_421" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_422" style="margin-left: 5px; ">19%</p>
         </td>
         <td colspan="1" id="_423" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_424" style="margin-left: 5px; ">255</p>
         </td>
         <td colspan="1" id="_506" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_507" style="margin-left: 5px; ">270</p>
         </td>
      </tr>
      <tr>
         <td colspan="1" id="_425" rowspan="1" style="width: 250px; vertical-align: center; ">
            <p dir="ltr" id="_426" style="margin-left: 5px; ">Create a small module artifact end-to-end scenario</p>
         </td>
         <td colspan="1" id="_427" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_428" style="margin-left: 5px; ">6%</p>
         </td>
         <td colspan="1" id="_429" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_430" style="margin-left: 5px; ">132</p>
         </td>
         <td colspan="1" id="_508" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_509" style="margin-left: 5px; ">133</p>
         </td>
      </tr>
      <tr>
         <td colspan="1" id="_431" rowspan="1" style="width: 250px; vertical-align: center; ">
            <p dir="ltr" id="_432" style="margin-left: 5px; ">Create a comment in a module artifact</p>
         </td>
         <td colspan="1" id="_433" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_434" style="margin-left: 5px; ">19%</p>
         </td>
         <td colspan="1" id="_435" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_436" style="margin-left: 5px; ">815</p>
         </td>
         <td colspan="1" id="_510" rowspan="1" style="width: 142px; vertical-align: center; ">
            <p dir="ltr" id="_511" style="margin-left: 5px; ">843</p>
         </td>
      </tr>
      <tr>
         <td colspan="1" id="_449" rowspan="1" style="width: 250px; vertical-align: top; ">
            <p dir="ltr" id="_450" style="margin-left: 5px; ">Hover over a module artifact and edit it</p>
         </td>
         <td colspan="1" id="_451" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_452" style="margin-left: 5px; ">2%</p>
         </td>
         <td colspan="1" id="_453" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_454" style="margin-left: 5px; ">87</p>
         </td>
         <td colspan="1" id="_513" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_514" style="margin-left: 5px; ">90</p>
         </td>
      </tr>
      <tr>
         <td colspan="1" id="_455" rowspan="1" style="width: 250px; vertical-align: top; ">
            <p dir="ltr" id="_456" style="margin-left: 5px; ">Hover over and open a collection</p>
         </td>
         <td colspan="1" id="_457" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_458" style="margin-left: 5px; ">2%</p>
         </td>
         <td colspan="1" id="_459" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_460" style="margin-left: 5px; ">108</p>
         </td>
         <td colspan="1" id="_515" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_516" style="margin-left: 5px; ">111</p>
         </td>
      </tr>
      <tr>
         <td colspan="1" id="_461" rowspan="1" style="width: 250px; vertical-align: top; ">
            <p dir="ltr" id="_462" style="margin-left: 5px; ">Manage folders</p>
         </td>
         <td colspan="1" id="_463" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_464" style="margin-left: 5px; ">2%</p>
         </td>
         <td colspan="1" id="_465" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_466" style="margin-left: 5px; ">72</p>
         </td>
         <td colspan="1" id="_517" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_518" style="margin-left: 5px; ">74</p>
         </td>
      </tr>
      <tr>
         <td colspan="1" id="_473" rowspan="1" style="width: 250px; vertical-align: top; ">
            <p dir="ltr" id="_474" style="margin-left: 5px; ">Open the project dashboard</p>
         </td>
         <td colspan="1" id="_475" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_476" style="margin-left: 5px; ">5%</p>
         </td>
         <td colspan="1" id="_477" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_478" style="margin-left: 5px; ">520</p>
         </td>
         <td colspan="1" id="_519" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_520" style="margin-left: 5px; ">534</p>
         </td>
      </tr>
      <tr>
         <td colspan="1" id="_479" rowspan="1" style="width: 250px; vertical-align: top; ">
            <p dir="ltr" id="_480" style="margin-left: 5px; ">Search by ID and string</p>
         </td>
         <td colspan="1" id="_481" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_482" style="margin-left: 5px; ">9%</p>
         </td>
         <td colspan="1" id="_484" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_485" style="margin-left: 5px; ">475</p>
         </td>
         <td colspan="1" id="_521" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_522" style="margin-left: 5px; ">492</p>
         </td>
      </tr>
      <tr>
         <td colspan="1" id="_486" rowspan="1" style="width: 250px; vertical-align: top; ">
            <p dir="ltr" id="_487" style="margin-left: 5px; ">Scroll 20 pages in a module</p>
         </td>
         <td colspan="1" id="_488" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_489" style="margin-left: 5px; ">19%</p>
         </td>
         <td colspan="1" id="_490" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_491" style="margin-left: 5px; ">308</p>
         </td>
         <td colspan="1" id="_523" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_524" style="margin-left: 5px; ">376</p>
         </td>
      </tr>
      <tr>
         <td colspan="1" id="_492" rowspan="1" style="width: 250px; vertical-align: top; ">
            <p dir="ltr" id="_493" style="margin-left: 5px; ">Switch the module view</p>
         </td>
         <td colspan="1" id="_494" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_495" style="margin-left: 5px; ">14%</p>
         </td>
         <td colspan="1" id="_496" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_497" style="margin-left: 5px; ">1002</p>
         </td>
         <td colspan="1" id="_525" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_526" style="margin-left: 5px; ">1056</p>
         </td>
      </tr>
      <tr>
         <td colspan="1" id="_498" rowspan="1" style="width: 250px; vertical-align: top; ">
            <p dir="ltr" id="_499" style="margin-left: 5px; ">Upload a 4 MB file as a new artifact</p>
         </td>
         <td colspan="1" id="_500" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_501" style="margin-left: 5px; ">4%</p>
         </td>
         <td colspan="1" id="_502" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_503" style="margin-left: 5px; ">157</p>
         </td>
         <td colspan="1" id="_527" rowspan="1" style="width: 142px; vertical-align: top; ">
            <p dir="ltr" id="_528" style="margin-left: 5px; ">164</p>
         </td>
      </tr>
   </tbody>
</table>

---+++Test case execution during test
#TestCases
<table class="gray-table">
	<tbody>
		<tr>
			<th align="center">Use case</th>
			<th align="center">% of Total Workload</th>
			<th align="center">Times executed in 5.0.2</th>
			<th align="center">Times executed in 6.0</th>
		</tr>
<!--		<tr>
			<td>Login</td>
			<td align="center">0</td>
  			<td align="center">386</td>
  			<td align="center">357</td>
		</tr>
-->
		<tr>
			<td>[[#CreateCollection][Create a collection]]</td>
			<td align="center">2</td>
			<td align="center">40</td>
			<td align="center">41</td>
		</tr>
		<tr>
			<td>[[#QueryByIdAndString][Query by ID and string]]</td>
			<td align="center">9</td>
			<td align="center">475</td>
			<td align="center">492</td>
		</tr>
		<tr>
			<td>[[#ManageFolders][Manage folders]]</td>
			<td align="center">2</td>
			<td align="center">72</td>
			<td align="center">74</td>
		</tr>
		<tr>
			<td>[[#CreateModArtifactComment][Add comments to a module artifact]]</td>
			<td align="center">19</td>
			<td align="center">815</td>
			<td align="center">843</td>
		</tr>
		<tr>
			<td>[[#OpenProjectDashboard][Open the project dashboard]]</td>
			<td align="center">5</td>
			<td align="center">520</td>
			<td align="center">534</td>
		</tr>
		<tr>
			<td>[[#CreateMedModArtifactE2E][Create a module artifact in a medium sized module]]</td>
 			<td align="center">6</td>
			 <td align="center">132</td>
			 <td align="center">133</td>
		</tr>
		<tr>
			<td>[[#CreateModArtifactE2E][Create a module artifact in a large module]]</td>
			<td align="center">19</td>
			<td align="center">255</td>
			<td align="center">270</td>
		</tr>
		<tr>
			<td>[[#CopyPaste25ModArtifacts][Copy and paste module artifacts in a large module]]</td>
			<td align="center">1</td>
			<td align="center">40</td>
			<td align="center">38</td>
		</tr>
		<tr>
			<td>[[#CreateArtifact][Create an artifact]]</td>
			<td align="center">3</td>
			<td align="center">121</td>
			<td align="center">125</td>
		</tr>
		<tr>
			<td>[[#HoverOpenCollection][Display the hover information for a collection]]</td>
			<td align="center">2</td>
			<td align="center">108</td>
			<td align="center">111</td>
		</tr>
		<tr>
			<td>[[#HoverEditModArtifact][Hover over module artifact]]</td>
			<td align="center">2</td>
			<td align="center">87</td>
			<td align="center">90</td>
		</tr>
		<tr>
			<td>[[#ScrollModule][Open a module and page down]]</td>
			<td align="center">19</td>
			<td align="center">308</td>
			<td align="center">376</td>
		</tr>
		<tr>
			<td>[[#SwitchModuleView][Change columns to display in a module]]</td>
			<td align="center">14</td>
			<td align="center">1002</td>
			<td align="center">1056</td>
		</tr>
<!--	<tr>
			<td>[[#DisplayModuleHistory][Display module history]]</td>
			<td align="center">1</td>
			<td align="center">1</td>
			<td align="center">9</td>
		</tr>
-->		<tr>
			<td>[[#Upload4MBFileInNewArtifact][Upload 4MB file in new artifact]]</td>
			<td align="center">4</td>
			<td align="center">157</td>
			<td align="center">164</td>
		</tr>
	</tbody>
</table>

---++++Response time comparison

The response time charts that follow use the median value (not the average).    The median is more consistent than the average in these tests, which makes it easier to compare response times between releases.   Response times are reported in milliseconds, where not otherwise specified.

In the repository that contained 517,080 artifacts with 250 concurrent
users, no obvious regression was shown when comparing response times
between runs.   A number of use cases have improved in 6.0.

---++Results
The chart below is an amalgam of all measurements from all use cases.    This indicates that performance is generally improved in 6.0.  

<img src="%ATTACHURLPATH%/Overview_Chart_5.png" width="60%" height="60%"/>
---++Observation
   * Memory and CPU utilization is similar in 6.0 as in 5.0.2, although there is slightly less CPU utilization on the RM server in 6.0.

<img src="%ATTACHURLPATH%/Overview_Chart_9.png"  width="60%" height="60%"/>
<img src="%ATTACHURLPATH%/Overview_Chart_15.png" width="60%" height="60%"/>

*Network traffic*
   * The following chart shows the amount of data sent over the network
<img src="%ATTACHURLPATH%/Overview_Chart_6.png" width="60%" height="60%" />

*Garbage collection*

Verbose garbage collection is enabled to create the GC logs.  The GC logs show very little variation between runs.  
The memory allocated to the Java heap is used in the following way:
<table class="gray-table">
	<tbody>
		<tr>
			<th>Version</th>
			<th>Nursery heap</th>
			<th>Tenured heap</th>
			<th>Cache</th>
		</tr>
		<tr>
			<td>RDNG 5.0.2</td>
			<td>4 GB</td>
			<td>10.8 GB</td>
			<td>1.2 GB</td>
		</tr>
		<tr>
			<td>RDNG 6.0</td>
			<td>4 GB</td>
			<td>8.9 GB</td>
			<td>3.1 GB</td>
		</tr>
	</tbody>
</table>
Below is an example of the output from the GC logs for JTS and RDNG, including versions 6.0 and 5.0.2 for each application server.

*RDNG*
   * Observation:  6.0 uses roughly 2G of additional Java heap space for caching.   Additional data is cached in memory as part of the 6.0 support for configuration management.
   * RDNG 6.0
<img src="%ATTACHURLPATH%/GC-RM60.png" width="50%" height="50%" />

   * RDNG 5.0.2
<img src="%ATTACHURLPATH%/GC-RM502.png" width="50%" height="50%" />

*JTS*
   * Observation: The graphs show similar behavior in 6.0 as in 5.0.2.  The activity is so small that there is not much to graph.

   * JTS 6.0
<img src="%ATTACHURLPATH%/GC-JTS60.png" width="50%" height="50%"/>

   * JTS 5.0.2
<img src="%ATTACHURLPATH%/GC-JTS502.png" width="50%" height="50%"/>

#CreateCollection 
---++++*Create a collection*
 
<img src="%ATTACHURLPATH%/All_pages_sorted_by_RM_scripts_Chart_21.png" width="70%" height="70%" />

[[#TestCases][Back to Test Cases & workload characterization]]

#QueryByIdAndString
---++++*Search an artifact using ID and a string*

<img src="%ATTACHURLPATH%/All_pages_sorted_by_RM_scripts_Chart_22.png" width="70%" height="70%"/>

[[#TestCases][Back to Test Cases & workload characterization]]

#ManageFolders
---++++*Create, move and delete folders*

<img src="%ATTACHURLPATH%/All_pages_sorted_by_RM_scripts_Chart_23.png" width="70%" height="70%" />

[[#TestCases][Back to Test Cases & workload characterization]]

#CreateModArtifactComment
---++++*Create a comment for a module artifact*
 
<img src="%ATTACHURLPATH%/All_pages_sorted_by_RM_scripts_Chart_25.png" width="70%" height="70%"/>

[[#TestCases][Back to Test Cases & workload characterization]]

#OpenProjectDashboard
---++++*Open the project dashboard*

<img src="%ATTACHURLPATH%/All_pages_sorted_by_RM_scripts_Chart_2.png" width="70%" height="70%"/>

[[#TestCases][Back to Test Cases & workload characterization]]

#CreateMedModArtifactE2E
---++++*Create a requirement in a medium sized module*

<img src="%ATTACHURLPATH%/All_pages_sorted_by_RM_scripts_Chart_26.png" width="70%" height="70%"/>

[[#TestCases][Back to Test Cases & workload characterization]]

#CreateModArtifactE2E
---++++*Create a requirement in a large module*

<img src="%ATTACHURLPATH%/All_pages_sorted_by_RM_scripts_Chart_27.png" width="70%" height="70%"/>

[[#TestCases][Back to Test Cases & workload characterization]]

#CopyPaste25ModArtifacts
---++++*Copy and paste 24 artifacts*

<img src="%ATTACHURLPATH%/All_pages_sorted_by_RM_scripts_Chart_28.png" width="70%" height="70%"/>

[[#TestCases][Back to Test Cases & workload characterization]]

#CreateArtifact
---++++*Create an artifact*

<img src="%ATTACHURLPATH%/All_pages_sorted_by_RM_scripts_Chart_29.png" width="70%" height="70%"/>

[[#TestCases][Back to Test Cases & workload characterization]]

#HoverOpenCollection
---++++*Display hover information for an artifact in a collection*

<img src="%ATTACHURLPATH%/All_pages_sorted_by_RM_scripts_Chart_30.png" width="70%" height="70%"/>

[[#TestCases][Back to Test Cases & workload characterization]]

#HoverEditModArtifact
---++++*Display hover information for a module artifact.  Open and edit artifact*

<img src="%ATTACHURLPATH%/All_pages_sorted_by_RM_scripts_Chart_42.png" width="70%" height="70%"/>

[[#TestCases][Back to Test Cases & workload characterization]]

#ScrollModule
---++++*Page down in a large module*

<img src="%ATTACHURLPATH%/All_pages_sorted_by_RM_scripts_Chart_31.png" width="70%" height="70%"/>

[[#TestCases][Back to Test Cases & workload characterization]]

#SwitchModuleView
---++++*Change display in a module*

<img src="%ATTACHURLPATH%/All_pages_sorted_by_RM_scripts_Chart_32.png" width="70%" height="70%"/>

[[#TestCases][Back to Test Cases & workload characterization]]

<!--
#DisplayModuleHistory
---++++*Display module history*

<img src="%ATTACHURLPATH%/All_pages_sorted_by_DNG_scripts_Chart_22.png" width="70%" height="70%"/>

[[#TestCases][Back to Test Cases & workload characterization]]
-->

#Upload4MBFileInNewArtifact
---++++*Uploading 4 MB file in new artifact*
The Save operation shows a small (200ms) regression in the runs below.    This operation has varied substantially from run to run, and there have been some 5.0.2 runs with substantially larger response times.    Our sense is that this is a "false regression", most likely caused by varying network conditions.
<img src="%ATTACHURLPATH%/All_pages_sorted_by_RM_scripts_Chart_33.png" width="70%" height="70%"/>

[[#TestCases][Back to Test Cases & workload characterization]]


---++ Appendix A
#AppendixA
<table class="gray-table">
	<tbody>
		<tr>
			<th align="left" width="200"><strong>Product</strong><br></th>
			<th align="left" width="100"><strong>Version</strong></th>
			<th align="left" width="600"><strong>Highlights for configurations under test</strong></th>
		</tr>
		<tr>
			<td style="vertical-align: top;">IBM HTTP Server for !WebSphere Application Server</td>
			<td style="vertical-align: top;">8.5.5.1</td>
			<td style="vertical-align: top;">IBM HTTP Server functions as a reverse proxy server implemented
via Web server plug-in for !WebSphere Application Server. Configuration details can be found from the [[http://pic.dhe.ibm.com/infocenter/clmhelp/v4r0/index.jsp?topic=%2Fcom.ibm.jazz.install.doc%2Ftopics%2Ft_config_reverse_proxy_ihs.html][CLM infocenter]].

*HTTP server (httpd.conf)*:

   * !MaxClients: increase value for high-volume loads [[http://publib.boulder.ibm.com/httpserv/ihsdiag/ihs_performance.html][(adjust value based on user load)]]
   * !ThreadsPerChild = 50

*Web server plugin-in (plugin-cfg.xml)*:

   * !ServerIOTimeout="900"

*OS Configuration*:

   * max user processes = unlimited
      </td>
		</tr>
		<tr>
			<td style="vertical-align: top;">IBM !WebSphere Application Server Network Deployment</td>
			<td style="vertical-align: top;">8.5.5.1</td>
			<td style="vertical-align: top;"><strong>JVM settings:</strong>

   * GC policy and arguments, max and init heap sizes:

				<verbatim>
-XX:MaxDirectMemorySize=1g -Xgcpolicy:gencon -Xmx16g -Xms16g -Xmn4g -Xcompressedrefs -Xgc:preferredHeapBase=0x100000000 -Xverbosegclog:logs/gc.log
</verbatim>

				*Thread pools:*

   * Maximum !WebContainer = 500
   * Minimum !WebContainer = 500


				*OS Configuration:*

				System wide resources for the app server process owner: 
   * max user processes = unlimited
   * open files = 65536

			</td>
		</tr>
		<tr>
			<td>DB2</td>
			<td>DB2 10.1</td>
			<td> </td>
		</tr>
		<tr>
			<td>LDAP server</td>
			<td> </td>
			<td> </td>
		</tr>
		<tr>
			<td>License server</td>
			<td> </td>
			<td>Hosted locally by JTS server</td>
		</tr>
		<tr>
			<td>RPT workbench</td>
			<td>8.3</td>
			<td>Defaults</td>
		</tr>
		<tr>
			<td>RPT agents</td>
			<td>8.5.1</td>
			<td>Defaults</td>
		</tr>
		<tr>
			<td>Network</td>
			<td> </td>
			<td>Shared subnet within test lab</td>
		</tr>
	</tbody>
</table>

---++++!! For more information
   * [[PerformanceDatasheetsAndSizingGuidelines#CLM_5_x][Performance Datasheets: CLM 5.x]]

---++++!! About the authors

Main.GustafSvensson

--------------------
---+++++!! Questions and comments:
   * What other performance information would you like to see here?
   * Do you have performance scenarios to share? 
   * Do you have scenarios that are not addressed in documentation?
   * Where are you having problems in performance?

%COMMENT{type="below" target="PerformanceDatasheetReaderComments" button="Submit"}%

%INCLUDE{"PerformanceDatasheetReaderComments"}%


<sticky></div></sticky>
Copyright © by IBM and non-IBM contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.