RTCEEWorkloadTests < Deployment

<div id="header-title" style="padding: 10px 15px; border-width:1px; border-style:solid; border-color:#FFD28C; background-image: url(<nop>https://jazz.net/wiki/pub/Deployment/WebPreferences/TLASE.jpg); background-size: cover; font-size:120%">
---+!! RTCEE Performance Testing: EE Workload Scenario<img src="https://jazz.net/wiki/pub/Deployment/WebPreferences/todo.png" alt="todo.png" width="50" height="50" align="right"> 
%DKGRAY% Authors: [[Main.JorgeAlbertoDiaz][Jorge Diaz]] <br>
Build basis: RTC 5.x
%ENDCOLOR%</div></sticky>

<!-- Page contents top of page on right hand side in box -->
<sticky><div style="float:right; border-width:1px; border-style:solid; border-color:#DFDFDF; background-color:#F6F6F6; margin:0 0 15px 15px; padding: 0 15px 0 15px;">
%TOC{title="Page contents"}%
</div></sticky>

<sticky><div style="margin:15px;"></sticky>
%STARTSECTION{"Intro"}%
This scenario tests the performance of the enterprise Dependency Based Build under high volume workload, simulating concurrent build operations that developers would execute in a real case scenario.<br/>
The tests itself are based on observations of the usage performed by our customers and our recommended practices to them.
%ENDSECTION{"Intro"}%
---++ Scenario Detailed Description
---+++ RTC Parametrization

---++++ Test assets configuration:

   * Personal builds requested will have the Minimum Load option set
   * Several streams are used in these tests. These streams have *not* been generated using the “Duplicate” stream feature. Duplicating a stream causes the Source Code Data to be reused, while for the tests information gathering it is preferred that every stream has different set of SCD.
   * System Definitions: 4.0.3 Link-edit feature will not be used.
   * Build Engines configuration: Build engines should be configured in such a manner that it is assured that a team build request doesn't impact personal builds concurrency. To accomplish that, the following property is added to the build engines that will be serving personal builds: “requestFilter” with value ”personalBuild=true”:

<img src="%ATTACHURLPATH%/buildFilter.png"/>

---++++ RTC Server configuration options:

   * Set advance property: 'Consider only outgoing change sets in personal builds = true'
   * Customization of the _Asynchronous Task Pool Size_ parameter for the amount of concurrent builds that will be executed.
   * Source Code Data Collection: customize the Max Number of Files in _Source Code Data Resource_ parameter to a maximum value of 100:

<img src="%ATTACHURLPATH%/scdConfig.png"/>

---+++ Virtual users characterization

To emulate the load and the main tasks expected during real solution usage, the following roles and operations have been identified. <br/>
These are used in the testing framework to emulate real users operations at the load and rates finer specified later in this topic:

   * Developer role A:
      * Change a COBOL program
      * Check-in the changes
      * Request Personal Build to verify changes
      * Accept incoming changes
      * Deliver his change sets

   * Developer role B:
      * Change a copybook source
      * Check-in the changes
      * Request Personal Build to verify changes
      * Accept incoming changes
      * Deliver his change sets

   * Contributor role:
      * Login to repository
      * Create and save a work item
      * Execute a query (25%) -- just execute query every 4 iterations
      * Logout

<br/>

---+++ Scenario execution preparation

Following are the main steps for preparing the environment for the test scenario run. The steps are beyond creating and configuring the assets already described:
   1. Assign/verify Build System definitions with sample resources
   1. Perform a full build of the applications
   1. Load the virtual developers workspaces in preparation for the test run.

<br/>

---+++ Scenario Variances

Based on the previous detailed configuration, two variances of the scenario are tested:

---++++ Test Scenario1

The following SCM structure is used:
   * 50 components distributed among 25 streams:
   * Each stream with 2 of the components with the test data sample distribution: one component will hold the “common” project of called “MortgageApplication-Common”,  where the other component contains the rest of projects

The following screenshot shows a sample configuration of one of the streams:

<img src="%ATTACHURLPATH%/streamConf1.png"/>
<img src="%ATTACHURLPATH%/streamConf2.png"/>

   * 30 build engines targeting the same build agent. This will allow the concurrency described later and the team build.
      * 25 build engines will be configured to just accept personal builds
   * One build definition for each stream representing the build of the test data application
      * Each stream will have a team build with associated:
         * Build workspace
         * Defined HLQ and load directory for every team / build definition
   * Each virtual user will need to have defined:
      * A personal repository workspace flowing to the assigned stream
      * A personal HLQ and load directory on the build machine
      * Personal workspace, HLQ and load directory will be used in the automated build requests processing

The scenario is meant to be performed during a long run of *8 hours*, trying to emulate a system load during a working day, with the following details:

   * The emulation will try to reproduce a scenario during a work day and variations of work load to different applications and streams. Similar to what would be normal lifecycle of development enterprise work, where some applications are more active than others.
      * From the 25 defined streams, 10 would be identified as more active ones, with users and load described in next subpoints.
      * Simulation of 28 concurrent developers.
      * Following distribution of emulated developers and streams:

2 streams will have 7 concurrent developers with the following roles and files to be modified in each iteration:

<table class="gray-table" style="width: 70%">
 <tr>
  <th><strong>Role</strong></th>
  <th><strong>#users</strong></th>
  <th><strong>Files to modify<br/>(1 line per developer)</strong></th>
 </tr>
  <tr>
  <td>Developer role A</td>
  <td>6 (per stream)</td>
  <td>AAACMORT.cbl / AABCMORT.cbl / AACCMORT.cbl<br/>
AADCMORT.cbl / AAECMORT.cbl / AAFCMORT.cbl<br/>
AAGCMORT.cbl / AAHCMORT.cbl / AAICMORT.cbl<br/>
AAJCMORT.cbl / AAKCMORT.cbl / AALCMORT.cbl<br/>
AAMCMORT.cbl / AANCMORT.cbl / AAOCMORT.cbl<br/>
AAPCMORT.cbl / AAQCMORT.cbl / AARCMORT.cbl </td>
 </tr>
</tr>
  <tr>
  <td>Developer role B</td>
  <td>1 (per stream)</td>
  <td>AAAMLIS.bms / AAAMTCOM.cpy / AABMLIS.bms / AABMTCOM.cpy </td>
 </tr>
</table>

3 streams will have 3 concurrent developers with the following roles and files to be modified in each iteration:

<table class="gray-table" style="width: 70%">
 <tr>
  <th><strong>Role</strong></th>
  <th><strong>#users</strong></th>
  <th><strong>Files to modify<br/>(1 line per developer)</strong></th>
 </tr>
  <tr>
  <td>Developer role A</td>
  <td>2 (per stream)</td>
  <td>AAACMORT.cbl / AABCMORT.cbl / AACCMORT.cbl<br/>
AADCMORT.cbl / AAECMORT.cbl / AAFCMORT.cbl
</td>
 </tr>
</tr>
  <tr>
  <td>Developer role B</td>
  <td>1 (per stream)</td>
  <td>AAAMLIS.bms / AAAMTCOM.cpy / AABMLIS.bms / AABMTCOM.cpy</td>
 </tr>
</table>


5 streams will have 1 developer assigned: 1:1 relationship between developers and streams:

<table class="gray-table" style="width: 70%">
 <tr>
  <th><strong>Role</strong></th>
  <th><strong>#users</strong></th>
  <th><strong>Files to modify<br/>(1 line per developer)</strong></th>
 </tr>
  <tr>
  <td>Developer role A</td>
  <td>1 (per stream)</td>
  <td>AAACMORT.cbl / AABCMORT.cbl / AACCMORT.cbl</td>
 </tr>
</table>

<br/>

Additionally, one team build will be requested at every hour for each of the streams: not scheduled to start at same hour every time but with a difference of 10 minutes., so not all the team builds are executed at same hour time.
---+++++ %BLUE%#Test Configuration%ENDCOLOR%
RTC EE workload scenario uses Mortgage Application x500 sample projects for the test:
 
   * Mortgage Application *500 which is 500 duplicates of the [[https://jazz.net/wiki/bin/view/Main/ZOSBuildSamplesV4][Mortgage sample application]]

<table class="gray-table" style="width: 60%">
 <tr>
  <th><strong>Test Data</strong></th>
  <th></th>
 </tr>
 <tr>
  <td><strong>Sample Project</strong></td>
  <td>MortgageApplication*500</td>
</tr>
  <tr>
  <td><strong>Assets</strong></td>
  <td>3000 COBOL programs<br/> 2000 Copybooks<br/> 1000 BMS<br/>3 others<br/>
  </td>
  </tr>
 <tr>
  <td><strong>Total Assets</strong></td>
  <td>6003
  </td>
</tr>
</table>

We use this configuration for RTC EE scenario 1 test:
<table class="gray-table">
 <tr>
  <th><strong>Test Environment</strong></th>
  <th>Configuration</th>
  <th>Model</th>
  <th>MSU</th>
 </tr>
 <tr>
  <td><strong>RTC Server</strong></td>
  <td>Operating System & Version: z/OS 01.12.00 </br>
          System Resource : 10 GB Storage, 2 CPs (1000 mips, CPU type : 2097.710) </br>
          CALM: 5.0 GA, 4 GB heap size </br>
          WAS: 8.5.5.1 </br>
  </td>
  <td>E12</td>
  <td/>
 </tr>
  <tr>
  <td><strong>Build Forge Agent & DB2z</strong></td>
  <td>Operating System & Version: z/OS 01.12.00</br>
         System Resource: 6 GB Storage, 4 CPs (2000 mips, CPU type : 2097.710) </br>
         Build System Toolkit: 5.0 GA </br>
		 DB2: 10.1.0.0 </br>
  </td>
  <td>E12</td>
  <td>513</td>
  </tr>
</table>

---+++++ %BLUE%#Test Report of Scenario1%ENDCOLOR%
   * For 8 hours long run of scenario1, 46 times(iterations) concurrent builds have been launched, total 1288 personal builds and 44 team builds have been processed successfully.
<table class="gray-table">
 <tr>
  <th>Start Time</th>
  <th>End Time</th>
  <th>Concurrent Request Times</th>
  <th>Concurrent Users Each Time</th>
  <th>Total Personal Builds Num</th>
  <th>Team Build Request Times</th>
  <th>Total Team Builds Num</th>
 </tr>
 <tr>
  <td>8/16/2014 11:30:50 AM</td>
  <td>8/16/2014 7:34:38 PM</td>
  <td>46</td>
  <td>28</td>
  <td>1288</td>
  <td>4</td>
  <td>40<td>
 </tr>
</table>
   * Personal build time/aversage build time of concurrent builds in 8 hours test;
<table class="gray-table">
 <tr>
  <th>Build Definition</th>
  <th>Build duration of Personal Build</th>
 </tr>
 <tr>
  <td>Build Definition 1 - 2</td>
  <td>22 seconds - 4 minutes 22 seconds, average 2 minutes<td>
 </tr>
 <tr>
  <td>Build Definition 3 - 5</td>
  <td>30 seconds - 4 minutes 53 seconds, average 2 minutes 30 seconds<td>
 </tr>
 <tr>
  <td>Build Definition 6 - 10</td>
  <td>29 seconds - 3 minutes 43 seconds, average 2 minutes 10 seconds<td>
 </tr>
</table>
   * No build(both personal and team build) fails in EE workload scenario 1 test;
   * %RED%CPU usage of CLM server is < 60% %ENDCOLOR%;<br/>
   <img src="%ATTACHURLPATH%/avganalysisOfclmserver.jpg"/>
   <img src="%ATTACHURLPATH%/cpudsad_clmserver.jpg">
   
   * %RED%CPU usage of BST & DB2z is < 60% %ENDCOLOR%;<br/>
   <img src="%ATTACHURLPATH%/avganalysisOfbst.jpg"/>
   <img src="%ATTACHURLPATH%/cpudsad_bst.jpg">
   
   * %RED%CPU usage of DB2z is smoothly under 6% %ENDCOLOR%;<br/>
   <img src="%ATTACHURLPATH%/cpuOfDB2z.jpg">
   
   * 140.4 Contributors are simulated in 2nd run - Overall Count Attempted Tasks For Run = 23990
   * Size of repository volume don't increases in 8 hours exeuction(compare end with start condition).
   * Indice incremental rate:
 <table class="gray-table">
 <tr>
  <th>Indice Directory Name</th>
  <th>Before execution</th>
  <th>After execution</th>
  <th>Incresed Rate</th>
 </tr>
 <tr>
  <td>_Indices</td>
  <td>2115620</td>
  <td>2242828</td>
  <td>6.01%</td>
 </tr>
 <tr>
  <td>./jfs-rdfhistory</td>
  <td>308</td>
  <td>424</td>
  <td>37.66%</td>
 </tr>
 <tr>
  <td>./jfs-rdfindex</td>
  <td>1986436</td>
  <td>2112940</td>
  <td>6.37%</td>
 </tr>
 <tr>
  <td>./jfs-texthistory</td>
  <td>52</td>
  <td>252</td>
  <td>0.00%</td>
 </tr>
 <tr>
  <td>./jfs-textindex</td>
  <td>128816</td>
  <td>129404</td>
  <td>0.46%</td>
 </tr>
</table>
---++++ Test Scenario2

The following SCM structure is used:
   * 150 components distributed among 25 streams:
   * Each stream with 6 components with the test data distribution, using a different set of qualified file names

The following screenshot shows a sample configuration of one of the streams:

<img src="%ATTACHURLPATH%/streamConf1.png"/>

   * 60 build engines targeting the same build agent. This will allow the concurrency described later and the team build.
      * 50 build engines will be configured to just accept personal builds
   * Three build definitions for each stream for building each set of the testing data
      * Each stream will have a team build with associated:
         * Build workspace
         * Defined HLQ and load directory for every team / build definition
   * Each virtual user will need to have defined:
      * A personal repository workspace flowing to the assigned stream
      * A personal HLQ and load directory on the build machine
      * Personal workspace, HLQ and load directory will be used in the automated build requests processing

The emulation will try to reproduce a scenario during a work day and variations of work load to different applications and streams. Similar to what would be normal lifecycle of development enterprise work, where some applications are more active than others: as Scenario 1 but emulating a system with heavier load:
      * From the 25 defined streams, 11 would be identified as more active ones, with users and load described in next subpoints.
      * Simulation of 75 concurrent developers.
      * Following distribution of emulated developers and streams:

3 streams will have 7 concurrent developers with the following roles and files to be modified in each iteration:

<table class="gray-table" style="width: 70%">
 <tr>
  <th><strong>Role</strong></th>
  <th><strong>#users</strong></th>
  <th><strong>Files to modify<br/>(1 line per developer)</strong></th>
 </tr>
  <tr>
  <td>Developer role A</td>
  <td>9 (per stream)</td>
  <td>AAACMORT.cbl / AABCMORT.cbl / AACCMORT.cbl<br/>
AADCMORT.cbl / AAECMORT.cbl / AAFCMORT.cbl<br/>
AAGCMORT.cbl / AAHCMORT.cbl / AAICMORT.cbl<br/>
AAJCMORT.cbl / AAKCMORT.cbl / AALCMORT.cbl<br/>
AAMCMORT.cbl / AANCMORT.cbl / AAOCMORT.cbl<br/>
AAPCMORT.cbl / AAQCMORT.cbl / AARCMORT.cbl<br/>
AAQCMORT.cbl / AARCMORT.cbl / AASCMORT.cbl<br/>
AATCMORT.cbl / AAUCMORT.cbl / AAWCMORT.cbl<br/>
AAXCMORT.cbl / AAYCMORT.cbl / AAZCMORT.cbl </td>
 </tr>
</tr>
  <tr>
  <td>Developer role B</td>
  <td>3 (per stream)</td>
  <td>AAAMLIS.bms / AAAMTCOM.cpy / AABMLIS.bms / AABMTCOM.cpy<br/>
AACMLIS.bms / AACMTCOM.cpy / AADMLIS.bms / AADMTCOM.cpy<br/>
AAEMLIS.bms / AAEMTCOM.cpy / AAFMLIS.bms / AAFMTCOM.cpy</td>
 </tr>
</table>

5 streams will have 6 concurrent developers with the following roles and files to be modified in each iteration:

<table class="gray-table" style="width: 70%">
 <tr>
  <th><strong>Role</strong></th>
  <th><strong>#users</strong></th>
  <th><strong>Files to modify<br/>(1 line per developer)</strong></th>
 </tr>
  <tr>
  <td>Developer role A</td>
  <td>5 (per stream)</td>
  <td>AAACMORT.cbl / AABCMORT.cbl / AACCMORT.cbl<br/>
AADCMORT.cbl / AAECMORT.cbl / AAFCMORT.cbl<br/>
AAGCMORT.cbl / AAHCMORT.cbl / AAICMORT.cbl<br/>
AAJCMORT.cbl / AAKCMORT.cbl / AALCMORT.cbl<br/>
AAMCMORT.cbl / AANCMORT.cbl / AAOCMORT.cbl
</td>
 </tr>
</tr>
  <tr>
  <td>Developer role B</td>
  <td>1 (per stream)</td>
  <td>AAAMLIS.bms / AAAMTCOM.cpy / AABMLIS.bms / AABMTCOM.cpy<br/>
AACMLIS.bms / AACMTCOM.cpy / AADMLIS.bms / AADMTCOM.cpy</td>
 </tr>
</table>


3 streams will have 3 developers assigned: 1:1 relationship between developers and streams:

<table class="gray-table" style="width: 70%">
 <tr>
  <th><strong>Role</strong></th>
  <th><strong>#users</strong></th>
  <th><strong>Files to modify<br/>(1 line per developer)</strong></th>
 </tr>
  <tr>
  <td>Developer role A</td>
  <td>3 (per stream)</td>
  <td>AAACMORT.cbl / AABCMORT.cbl / AACCMORT.cbl</td>
 </tr>
</table>

<br/>

Repeating the role steps at cycles of 30 minutes.
Additionally, one team build will be requested at every hour for each of the streams: not scheduled to start at same hour every time but with a difference of 10 minutes., so not all the team builds are executed at same hour time.



---++ Test Topologies

The following topologies are being used for the execution of this scenario in order to test the performance and gather relevant data.

%INCLUDE{"RTCEETestingTopologies" section="DualTierZ"}%

---++ Test Data

From the existing scalability test data based on [[https://jazz.net/wiki/pub/Main/ZOSBuildSamplesV4/MortgageApplication.zip][Mortage Application]] sample, the following will be used:
%INCLUDE{"RTCEEPerfTestsData" section="Mortx500"}%

---++ Test Reports

%RED% Fill with a brief introduction and links to relevant data %ENDCOLOR%

---+++++!! Related topics:

---+++++!! External links:          

   * [[https://www.ibm.com][IBM]]

---+++++!! Additional contributors:
     

<sticky></div></sticky>
This topic: Deployment > RwattsSandbox > DeploymentPlanningAndDesign > PerformanceDatasheetsAndSizingGuidelines > RTCSystemZPerformanceTesting > RTCEEWorkloadTests
History: r2 - 2014-10-22 - 09:04:05 - Main.rucielulu
Copyright © by IBM and non-IBM contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.