RequirementsManagement70Performance < Deployment

Engineering Lifecycle Management Wiki - Deployment
Community information and contribution guidelines
TWiki > Deployment Web > DeploymentPlanningAndDesign > PerformanceDatasheetsAndSizingGuidelines > RequirementsManagement70Performance
Revision 58 - 2025-08-02 - 07:10:38 - PaulEllis
<div id="header-title" style="padding: 10px 15px; border-width:1px; border-style:solid; border-color:#FFD28C; background-image: url(<nop>https://jazz.net/wiki/pub/Deployment/WebPreferences/TLASE.jpg); background-size: cover; font-size:120%">
---+!! 7.0 performance:   IBM Engineering Requirements Management DOORS Next 
%DKGRAY% Authors: Main.VaughnRokosz, Main.SkyeBischoff, Main.HongyanHuo, Main.LeeByrnes </br>
Build basis: 7.0
%ENDCOLOR%</div></sticky>

<!-- Page contents top of page on right hand side in box -->
<sticky><div style="float:right; border-width:1px; border-style:solid; border-color:#DFDFDF; background-color:#F6F6F6; margin:0 0 15px 15px; padding: 0 15px 0 15px;">
%TOC{title="Page contents"}%
</div></sticky>

<sticky><div style="margin:15px;"></sticky>

---++ Introduction

This article presents the results of the performance testing conducted against the 7.0 release of IBM Engineering Requirements Management DOORS Next.   This release greatly improves scalability and reliability, increasing the number of artifacts you can manage on a single server by almost an order of magnitude.
  
Here is what you can expect from 7.0:
   * Support for up to 20 million artifacts on Oracle (10 million on DB2) - up from 2.3 million in 6.x.
   * Support for global configurations containing up to 2500 contributions on Oracle (1250 on DB2).
   * Higher user loads (up to 500 users) with less memory required on the DNG server.
   * More resilience under stress (no more "rogue queries").

We'll cover the following topics in this article:
   * [[#DataLimits][Data limits in 7.0]]
   * [[#DataShapeAndWorkload][Changes to the data shape and workload for 7.0 testing]]
   * How much memory to allocate to your database server for different repository sizes ([[#DatabaseMemoryOracle][Oracle]] and [[#DatabaseMemoryDb2][DB2]]).
   * [[#GlobalConfSize][How the number of contributions in a global configuration impacts performance]]
      * [[#GlobalConfSizeOracle][Oracle]]
      * [[#GlobalConfSizeDB2][DB2]]
   * [[#ArtifactCount][How the total number of artifacts impacts performance]]
      * [[#ArtifactCountOracle][Oracle]]
      * [[#ArtifactCountDB2][DB2]]
   * [[#TwentyMillion][Test results for a 20 million artifact repository (Oracle)]]
   * [[#TenMillionDB2][Test results for a 10 million artifact repository (DB2)]]
   * [[#TenMillion][Test results for a 10 million artifact repository (Oracle)]]
   * [[#UserLoad][What happens as the user load is increased past 500 users?]]
   * [[#ProjectSize][How project area size impacts performance]]

If you want a bit more background as to what's changed in the release to support higher scale, you can jump to a [[#ArchitectureChanges][summary of architectural changes]].

---+++!! Disclaimer

%INCLUDE{"PerformanceDatasheetDisclaimerEndToEnd"}%

#DataLimits
---+++ Data limits in 7.0

The table below summarizes the data scale limits for 7.0.   You'll get the best performance by organizing your artifacts into many small components. </br>
(Updated June 2022): Also note: [[https://jazz.net/wiki/bin/view/Deployment/PerformanceDoorsNext702iFix8][ 7.0.2 iFix008 Performance Update: IBM Engineering Requirements Management DOORS Next]]

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="right" disableallsort="on" tablewidth="40%" }%
|		|	Oracle	|	DB2	|	SQL Server*	|
|	Repository size (artifact versions)	|	20,000,000	|	10,000,000	|	2,300,000	|
|	Project area size (artifact versions)	|	3,800,000	|	2,000,000	|	200,000	|
|	Total number of components	|	4000	|	2500	|	500	|
|	[[RequirementsManagement70Performance#Global_configurations][Max contributions in a GC]] 	|	2500	|	1250	|	25	|
|	Components per project	|	1000	|	215 |	100	|
|	Artifacts per module	|	10000	|	10000 |	10000	|
|	Artifacts per folder	|	10000	|	10000 |	10000	|
|	Artifacts returned by a view	|	10000	|	10000 |	10000	|
<br>
Repository and project area sizes are expressed in terms of the number of rows in the DNG artifacts table. In end-user terms, this would be the number of versions for requirements and modules.  For example, when we say that Oracle can support 20 million artifact versions, we mean that the number of rows in the DNG artifacts table can be up to 20 million. You can get the artifact count from an existing repository by running a SQL query:
<verbatim>
Oracle:   select count(*) from dngartifacts_db_artifact;
Db2: select count(*) from dngartifacts.db_artifact;
</verbatim>
A more detailed breakdown of the artifacts in the test environment can be found in the [[RequirementsManagement70Performance#Artifact_counts][Data shapes]] section.

<br>
Notes (Updated June 2022):
   * *<b>At the time of GA (General Availabilty) of 7.0 "We did not do performance testing with SQL Server in the 7.0 release, so we are not changing the recommended limits for SQL Server in 7.0."
   * There have been additional updates on SQL Server performance where these benchmarks were invalidated and more realistic benchmarks were published. 
      [[https://www.ibm.com/support/pages/node/6491169][Thresholds for determining whether your DOORS Next instance has exceeded the capacity of Microsoft® SQL Server.]]</b>
   * These are not hard limits.   However, if you grow past these limits, you may see response times increasing past 5 seconds.
   * DB2 performance degrades as the size of the global configuration increases.    Response times for some operations were high (more than 5 seconds) when running with a GC of 1250 on DB2, in a repository containing 10 million artifacts.
   * For the Oracle sizing limits above, Oracle 19c was used.

#DataShapeAndWorkload
---+++ Changes to the data shape and workload for 7.0 performance tests

We have greatly increased the complexity of data used in 7.0 testing, as compared with performance tests on prior releases.   This was in part motivated by the order of magnitude improvement in the number of supported artifacts in 7.0, but it was also motivated by customer feedback on data shape, artifact counts, and use cases.    This testing is the most complex configuration attempted so far in ELM, and reflects typical usage patterns for fine-grained components in global configurations.

The changes include:
   * All use cases execute within the context of global configurations.
   * Global configurations in 7.0 are 100 times larger than in previous tests (up to 2500 contributions)
   * Use cases include link creation and views with link columns
   * Editing simulations involve more changes  (50 changes per deliver vs 1 change in 6.x)
   * Artifacts are organized into many small components, with 4 times more components used in 7.0 (2100 vs. 500 in 6.x testing)
  
---+++ Architectural changes

#ArchitectureChanges

For 7.0, we have greatly increased the number of artifacts that could be managed by a single server, and we have improved the resiliency of the server when under stress.  These improvements were made possible by exploiting the capabilities of the relational database server.

Prior to the 7.0 release, DOORS Next stored information about requirements in an index that lived on the DOORS Next server.   This index was built using a schema-less, triple-based technology called [[https://jena.apache.org/][Jena]].    Jena allowed the DOORS Next server to query against the index in a flexible way using a semantic query language called SPARQL.   Unfortunately, Jena became a bottleneck as deployments grew in size, and the limitations of Jena were the reason that 6.x and prior releases were limited to 2.3 million requirements or less.    Some of the specific problems with Jena included:
   * High memory requirements.    Jena performs best when the entire index is cached in memory.    This often meant that a DOORS Next server would need 128G or more of RAM.
   * Inefficient query scoping.    Jena does little optimization of SPARQL queries, and so some queries would end up processing far more artifacts than necessary.   This was especially true for queries involving links, or views built in the "All artifacts" section of the UI.   Even queries that returned only a few results might need to iterate over millions of irrelevant hits, leading to query timeouts.
   * Concurrency issues leading to server instability.    Jena uses a many-reader, one-writer model.   While read queries are in progress, write transactions are stored in a journal.    The journal grows until there is a quiet period when there are no reads in progress, and at that point, the journal is replayed into the main Jena index.   The overhead of the journal gets higher over time, and so if there are very long read transactions (e.g. because of inefficient scoping) or if there are just lots of shorter queries always running (e.g. because there are lots of users), performance gets steadily worse.    We refer to this as the Jena read/write bottleneck.

We addressed as many of the limitations as we could in 6.0.6.1, but it became clear to us that we could go no further with Jena.   So, in the 7.0 release, the product was re-implemented to rely on the relational database instead of the Jena index and to use SQL instead of SPARQL.   This has a number of advantages:
   * *Lower memory requirements for the DNG server*.    Our tests used systems with 32G of RAM, and this was sufficient for 500 user workloads executing against repositories with 20,000,000 requirements.   DOORS Next is now similar to the other ELM applications in terms of its system requirements.
   * *Sophisticated SQL query optimization*.   Relational databases have sophisticated optimizers which minimize the number of rows that need to be processed, even for complex joins, and database indexes provide additional optimizations.   The net result of using SQL is greatly improved scoping so that queries iterate through the smallest possible set of rows.
   * *Improved stability under load*.   Relational databases like DB2 and Oracle are designed for parallel processing from the ground up, unlike Jena.   In 7.0, there is no equivalent of a rogue query that will bring down the server, as was possible with Jena.   

One consequence of the new architecture is that work has shifted from the DOORS Next server to the database server.   You may need to allocate [[#DatabaseMemoryOracle][more memory to the database server]], or ensure that you have a high-speed storage subsystem.    In any case, you'll want to monitor the database after upgrading to 7.0 to see whether any tuning is required, and you'll want to plan for adding capability to your database server in order to exploit the higher levels of data scale that 7.0 makes possible.

#TenMillion

---++ Results from testing 10 million artifacts on Oracle 

This section summarizes the results of a 500 user test against an Oracle configuration with 10 million artifacts, using a global configuration with 2500 contributions.

In these tests, most operations complete in under 1 second;   the exceptions are listed below.

General observations:
   *  JTS CPU usage is negligible
   * Oracle CPU usage is low, while disk utilization is moderate
   * RM CPU usage is moderate

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="right" disableallsort="on" tablewidth="40%" }%
|	Operation	|	Time (ms)	|
|	Add link columns to a view	|	4,288	|
|	Hover over a link in the "Link to" column of a view	|	4,127	|
|	Select "OK" to discard a changeset with 50 changes	|	2,473	|
|	Hover over an artifact	|	2,235	|
|	Show module history	|	2,214	|
|	Select "OK" to discard a changeset with 10 changes	|	2,191	|
|	Open Module (10000 artifacts)	|	2,015	|
|	Select the "Artifact Links" tab in an artifact	|	1,905	|
|	Insert a new artifact into a view	|	1,815	|
|	!Upload a 4 MB file	|	1,550 |
|	Select the "Discard a changeset" menu	|	1,466	|
|	Select "OK" to create a link	|	1,347	|
|	Open Module (1500 artifacts)	|	1,208	|
|	Select a module in the "Create links" dialog	|	1,063	|
|	Select "OK" to create a changet set|	1,024	|
<br>

Some of the more complex operations (creating streams or baselines, delivering change sets, or creating reports from modules) can take longer - response times for the complex operations are listed below.

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="right" disableallsort="on" tablewidth="45%" }%
|	Operation	|	Time (ms)	|
|	Create a Word document from a module	|	33,605	|
|	Create a stream	|	16,263	|
|	Select "Deliver" for a change set that includes 10 links	|	15,178	|
|	Select "Deliver" for a change set that includes 50 changes	|	11,049	|
|	Select "Deliver" for a change set that includes 10 changes	|	10,834	|
|	Finish delivering a change set that includes 10 links	|	7,999	|
|	Finish delivering a change set that includes 10 changes|	7,145	|
|	Paste multiple artifacts into a module	|	7,111	|
|	Finish delivering a change set that includes 50 changes	|	7,066	|
|	Create a baseline |	4,266	|
<br>

About this test:
   * Number of simulated users:   500
   * [[#TestEnv2][Test environment 2]]
   * [[#WorkloadSummary][Standard performance workload]]
   * RM server CPU:   23%
   * JTS server CPU:  .4%
   * Oracle CPU:   9.8%
   * Oracle disk busy:   53%
   * Oracle reads per second:   268
   * Oracle MEMORY_TARGET:   48g
   * Number of requirements:  9,447,009
   * Number of versions:  58,106,678
   * Number of contributions in a global configuration:   2500
   * jazz.net work item:  [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/500817][500817: Oracle 10m (SoftLayer)]]

#TenMillionDB2
---++ Test results for a 10 million artifact repository (DB2)

This section summarizes the results of a 500 user test against a DB2 configuration with 10 million artifacts, using a global configuration with 1250 contributions.  This configuration is the maximum recommended size for DB2 in 7.0.    

General observations
   * CPU usage on the DNG server is moderate
   * Disk usage on the DB2 server is low
   * CPU usage on the JTS server is negligible
   * Please refer to the [[#Db2Tuning][DB2]] and [[#DngTuning][DNG]] tuning sections - some configuration parameters need to be adjusted to achieve this level of performance.

Common browse and edit operations complete in under 1 second, with the following exceptions:

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="right" disableallsort="on" tablewidth="40%" }%
|	Operation	|	Time (ms)	|
|	Add links columns to a module view	|	5526	|
|	Insert an artifact into a module 	|	2844	|
|	Show module history	|	5466	|
|	Click on "Discard a change set" to discard 10 changes	|	4640	|
|	Click on "Discard a change set" to discard 50 changes	|	5136	|
|	Select "OK" in the Create Links dialog	|	2110	|
|	Open a random module	|	1615	|
|	Open a module (1500 artifacts)	|	1608	|
|	Select "OK" in the Upload File dialog	|	2761	|
|	Open a module (500 artifacts)	|	1429	|
|	Open a module (10,000 artifacts)	|	3162	|
|	Hover over a link in a module view	|	2024	|
|	Select "Done" after editing an artifact	|	2394	|
|	Select "Create" in the Create Change set dialog	|	1528	|
|	Select a random artifact in a module	|	1423	|
|	Hover over an artifact in a module view	|	1538	|
|	Save a comment	|	1709	|
|	Select the "Create->Upload Artofact…" menu	|	1325	|
|	Select the target folder for an upload	|	1322	|
|	Open the configuration menu	|	1084	|
<br>

The more complex operations (creating streams or baselines, delivering change sets, or creating reports from modules) can take longer - response times for the complex operations are listed below. 

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="right" disableallsort="on" tablewidth="50%" }%
|	Operation	|	Time (ms) |
|	Create a Word document from a module	|	79887 |
|	Create a stream	|	19272 |
|	Select "Deliver Change Set" for a change set with 10 links	|	14920 |
|	Select "Deliver Change Set" for a change set with 10 changes	|	11240 |
|	Select "Deliver Change Set" for a change set with 50 changes	|	11007 |
|	Finish delivering a change set that includes 10 links	|	9312 |
|	Copy/paste for multiple artifacts in a module	|	9603 |
|	Finish delivering a change set that includes 50 changes	|	8503 |
|	Finish delivering a change set that includes 10 changes	|	8510 |
|	Create a baseline	|	4021 |
<br>

About this test:
   * [[#TestEnv4][Test environment 4]]
   * [[#Workload][Standard performance workload]]
   * Number of contributions in a global configuration:  1250
   * % disk busy on the DB2 server:   20%
   * DNG server CPU usage:  64%.    Note that CPU usage is higher in this test environment because the DNG server has only 16 processors.    The other test environments use 32 processors.
   * Repository size:  10 million artifacts (see [[#DataShape][data shape details]])
   * Additional details on jazz.net:   [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/502467][10m DB2:   additional runs with tuning]]

#TwentyMillion

---++ Results from testing 20 million artifacts on Oracle 

This section summarizes the results of a 500 user test against a configuration with 20 million artifacts.   Common browse and edit operations complete in under 1 second;   the exceptions are listed below.

General observations:
   *  JTS CPU usage is negligible
   * Oracle CPU usage is low, while disk utilization is moderate (63%)
   * RM CPU usage is moderate (under 25%)

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="right" disableallsort="on" tablewidth="35%" }%
|	Operation	|	Time (ms)	|
|	Add link columns to a view	|	2920	|
|	Discard a changeset containing 50 changes|	2581	|
|	Hover over a link in the "Link to" column of a view	|	2432	|
|	Open a module (10,000 artifacts)	|	2151	|
|	Show module history	|	2045	|
|	Upload a 4 MB file	|	1579	|
|	Insert an artifact into a module  view|	1570	|
|	Discard a changeset containing 10 changes|	1458	|
|	Hover over an artifact	|	1410	|
|	Select "OK" to create a change set	|	1236	|
|	Select "OK" to create a link	|	1197	|
|	Select the "Artifact Links" tab in an artifact	|	1092	|
|	Open a module (200 artifacts)  |	1019	|
<br>

Some of the more complex operations (creating streams or baselines, delivering change sets, or creating reports from modules) can take longer - response times for the complex operations are listed below.

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="right" disableallsort="on" tablewidth="45%" }%
|	Operation	|	Time (ms)	|
|	Create a Word document from a module	|	34006	|
|	Create a stream	|	20187	|
|	Select "Deliver" for a change set that includes 10 links	|	15393	|
|	Select "Deliver" for a change set that includes 10 changes |	11112	|
|	Select "Deliver" for a change set that includes 50 changes|	11104	|
|	Finish delivering a change set that includes 10 links	|	8281	|
|	Paste multiple artifacts into a module	|	7812	|
|	Finish delivering a change set that includes 50 changes	|	7323	|
|	Finish delivering a change set that includes 10 changes |	7315	|
|	Create a baseline	|	4560	|
<br>

About this test:
   * Number of simulated users:   500
   * [[#TestEnv2][Test environment 2]]
   * [[#WorkloadSummary][Standard performance workload]]
   * RM server CPU:   22%
   * JTS server CPU:  .4%
   * Oracle CPU:   6.7%
   * Oracle disk busy:   63%
   * Oracle reads per second:   327
   * Oracle MEMORY_TARGET:   48g
   * Number of requirements:  18,756,265
   * Number of versions: 118,928,959
   * Number of contributions in a global configuration:   2500
   * jazz.net work item:  [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/500766][500766: Oracle 20m tests (SoftLayer)]]

---++ Database memory requirements

#DatabaseMemoryOracle
---+++ Oracle memory vs. repository size

We ran a series of tests to determine how much Oracle memory was required for different repository sizes.    Here's a summary of what we found.
   * Larger deployments require a larger Oracle buffer cache.    A good rule of thumb is to set the buffer cache size to be 5-10% of the size of the RM tablespace.
   * Performance is better with large buffer caches, but there is a point of diminishing returns.   Beyond that point, the amount of improvement is small.
   * When the buffer cache is too small, page response times degrade significantly.   Disk utilization on the database server can reach 100%.
   * A deployment with 2 million artifacts required a buffer cache of 5G;   a deployment with 20 million artifacts required a buffer cache of 25G.   There is a linear relationship between tablespace size and optimal buffer cache size.

The graph below shows the optimal buffer cache size as a function of the size of the RM tablespace.   Larger tablespaces require larger buffer caches.    Oracle uses the buffer cache to avoid having to read from disk (since disk reads are slower than memory reads).   If there is more data on disk, then the memory cache needs to be larger to keep disk I/O low.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/OramemByTablespace.png" alt="OramemByTablespace.png"> 

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="right" disableallsort="on" tablewidth="30%" }%
| *Artifacts (millions)* | *RM tablespace (G) * | *Buffer cache (G)* | *% of tablespace size* |
| 2 | 49  | 5 | 10.2 |
| 5 | 118  | 7.5 | 6.4 |
| 10 | 221  | 16.7 | 7.6 |
| 15 | 362  | 20.3 | 5.6 |
| 20 | 460  | 25.2 | 5.6 |
<br>

The following graph shows the relationship between the number of requirements and the size of the RM tablespace for our [[#DataShape][specific datashape]].
<br>

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/RepoSizes.png" alt="RepoSizes.png"> 

---++++  Methodology

There is a certain amount of judgement involved in determining the optimal buffer cache size.    This section will describe the test and analysis methodology.   These tests were conducted on [[#TestEnv1][test environment 1]] and [[#TestEnv2][test environment 2]].   The results were the same for both test environments.

We ran the [[#WorkloadSummary][standard 500 user workload]] for several different settings of the Oracle MEMORY_TARGET parameter (Oracle's automatic memory management was enabled).    We collected operating system metrics during the test, and also captured an AWR report for a 1 hour period (at peak load).

For each run, we  extracted the number of Read IO requests per second and the average buffer cache size from the AWR report (see screenshots below).

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/awr_io.png" alt="awr_io.png"> 
<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/awr_cache.png" alt="awr_cache.png"> 

We then graphed the buffer cache and read I/O values.    

We also looked at response times and disk utilization.     We used the following set of guidelines to determine the optimal buffer cache size:
   * Disk utilization of under 75%
   * No significant degradation of response times
   * 500 read operations per second or less 

The optimal buffer cache size is where the curve starts to flatten out (where there are diminishing returns from adding more memory).    Here is what that graph looks like for the 5 million artifact repository.   

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/BufferCacheExample.png" alt="BufferCacheExample.png"> 

Here's an example of how response times vary as the buffer cache is increased.   For this use case, the point of diminishing returns is 5.4G (compared with 7.5G in the I/O graph).   In cases like this where the I/O and response time graphs gave different results, we used the larger of the values to be conservative.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/PageRespVCache.png" alt="PageRespVCache.png"> 

---++++  Additional information

The [[https://docs.oracle.com/en/database/oracle/oracle-database/19/tgdba/tuning-database-buffer-cache.html#GUID-F2D11BC8-986F-4A8B-A37F-530BE9EB2DA8][Oracle Database Performance Tuning guide]] discusses buffer cache tuning.   

For more details on the test runs, please refer to the following work items on jazz.net:

   * [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/499578][499578: 2m Oracle SL - tests to vary Oracle memory]]
   * [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/499876][499876: 5m Oracle SL - tests to vary Oracle memory]]
   * [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/500488][500488: 10m Oracle RTP - tests to vary Oracle memory]]
   * [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/500487][500487: 15m Oracle SL - tests to vary Oracle memory]]
   * [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/497119][497119: 20m Oracle SL - test runs to vary Oracle memory]]

#DatabaseMemoryDb2
---+++ DB2 memory vs. repository size

We ran a series of tests to determine how much DB2 memory was required for different repository sizes.    Here's a summary of what we found.
   * Larger deployments require a larger bufferpool.    A good rule of thumb is to set the bufferpool to be 10% of the size of the RM database.
   * Performance is better with large bufferpool, but there is a point of diminishing returns.   Beyond that point, the amount of improvement is small.
   * When the bufferpool is too small, page response times degrade significantly.   Disk utilization on the database server can reach 100%.
   * A deployment with 2 million artifacts required a bufferpool of 4G;   a deployment with 10 million artifacts required a bufferpool of 14G.   There is a linear relationship between tablespace size and optimal bufferpool size.
   * The bufferpool size is not dependent on the number of contributions in a global configuration.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/DB2Memory.png" alt="DB2Memory.png"> 

These tests were conducted on [[#TestEnv4][test environment 4]]. 

To find the optimal bufferpool size, we ran the [[#WorkloadSummary][standard 500 user workload]] for several different bufferpool settings, and collecting disk utilization data for each run.    We graphed the disk busy % against bufferpool size, and found the point in the curve where it starts to flatten out.    The bufferpool size at that point is the optimal size.   

The results for different repository sizes (using a global configuration with 250 contributions) is shown below.   The "red zone" is where we see degraded page performance.   In the yellow zone, the page performance is not impacted but disk utilization is high.   Bufferpools in the yellow zone may not sustain peak loads.    The green zone is recommended for optimal performance (4G for 2 million, 10G for 5 million, and 14G for 10 million).

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/DB2Bufferpool.png" alt="DB2Bufferpool.png"> 

For more details on the test runs that were used to construct these charts, please refer to the following work item on jazz.net:    [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/500494][DNG 10Mi Small and Medium GCs -  DB2 performance vs Database_Memory, bufferpool, and disk caching]].

#GlobalConfSize
---++ Impact of global configuration size

As a general rule, response times increase as the number of contributions to a global configuration (GC) increases.  This increase is small on Oracle, but is more significant on DB2.   Based on the tests described below, we are recommending that you limit your global configurations to 250 contributions or less on DB2.

#GlobalConfSizeOracle
---+++ Impact of global configuration size on Oracle

Response times increase as the number of contributions to a global configuration (GC) increases, but this increase in response time is often small - 1 second or less.  Additionally, the CPU utilization on the DNG server increases as GC size increases.   

You can see this trend in the chart below for the common browse and edit operations, for tests in a repository with 10 million artifacts.   This chart compares response times for operations executed in global configurations that contain 250, 1250, and 2500 contributions.    This has been filtered to show only those operations that degraded by more than one second when comparing the 250 and 2500 GCs.     But the pattern is consistent across most operations.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/GCSizeImpact.png" alt="GCSizeImpact.png"> 

Opening views that include link columns are impacted the most.   This is because the SQL queries to look up links must join across all of the streams that are part of the global configuration.   

The more complex operations (creating streams or baselines, delivering change sets, or creating reports from modules) are less sensitive to GC size (see chart below).   

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/GCSizeImpactTransactions.png" alt="GCSizeImpactTransactions.png"> 

CPU utilization for the 3 GC sizes is shown below.   More CPU is used on the DNG server for larger GCs.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/GCCPUImpact.png" alt="GCCPUImpact.png"> 

About this test:
   * [[#TestEnv3][Test environment 3]]
   * [[#WorkloadSummary][Standard performance workload]]
   * Repository size:  10 million artifacts (see [[#DataShape][data shape details]])
   * Additional details on jazz.net:   [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/499578][499620: Oracle 10 million repository tests]]


#GlobalConfSizeDB2
---+++ Impact of global configuration size on DB2

When using DB2, the number of contributions in a global configuration has a large impact on response times.   A few of the use cases degrade by 2-3 seconds when comparing GCs with 250 contributions to GCs with 1250 contributions.   There can be an additional degradation of 1 second when comparing GCs with 1250 contributions to those with 2500 contributions.    CPU usage on the RM server increases as the GC size increases.   Note that the [[#DngTuning][DNG server]] and the [[#Db2Tuning][DB2 server]] needed to be tuned to get these results.  The configuration cache size should be increased for the DNG server, and auto runstats should be disabled on the DB2 server.   

We recommend keeping the number of GC contributions under 1250 when using DB2.   And you should be aware that there are a few use cases that will be slow at 1250 and above (views with link columns, inserting an artifact into a module).

Here is a comparison for common browse and edit operations.   The chart includes operations that exceed 1 second; this is about 30% of the measured operations.    For the operations that are under 1 second, the pattern is similar, but the degradation is smaller (250ms - 500ms).   

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/GCSizeImpactDB2.png" alt="GCSizeImpactDB2.png"> 

The more complex operations (creating streams or baselines, delivering change sets, or creating reports from modules) are less sensitive to GC size (see chart below).    The one exception is creating a Word document from a module, which increases by 40 seconds when executed in a GC with 1250 contributions (and takes 180 seconds when executed in a GC with 2500 contributions).

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/GCSizeImpactTransDB2.png" alt="GCSizeImpactTransDB2.png"> 

The CPU usage on the DNG server increased from 53% (250 contribution GC) to 64% (1250 contribution GC).   The DB2 server load did not increase.

About these tests:
   * [[#TestEnv4][Test environment 4]]
   * [[#Workload][Standard performance workload]]
   * Repository size:  10 million artifacts (see [[#DataShape][data shape details]])
   * Additional details on jazz.net:   [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/502467][10m DB2:   additional runs with tuning]]

#ArtifactCount
---++ Impact of total artifact count

This section looks at how the number of artifacts impacts response times for our [[#WorkloadSummary][standard 500 user workload]].

#ArtifactCountOracle
---+++ Impact of total artifact count on Oracle

As the number of artifacts grows, you must increase the size of the Oracle buffer cache, as [[#DatabaseMemoryOracle][described in an earlier section]].   For a system with sufficient memory, there is a slight increase in response times across most use cases as the repository grows, but performance is not that sensitive to repository size.    The majority of the operations remain under 1 second even for a repository with 20 million artifacts, and the difference in times between a 2 million and a 20 million artifact repository is typically less than 1 second.

The following chart compares the response times of common browse and edit operations across different repository sizes.    These operations use a GC with 250 contributions.   The results have been filtered to include only those operations that take more than 1 second.    

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ArtifactCountImpact.png" alt="ArtifactCountImpact.png"> 

The more complex operations (creating streams or baselines, delivering change sets, or creating reports from modules) are not very sensitive to the repository size (see chart below).   The exception is stream creation, which does get slower as the number of artifacts increases.   

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ArtifactCountImpactTrans.png" alt="ArtifactCountImpactTrans.png"> 

About these tests:
   * [[#TestEnv2][Test environment 2]]
   * [[#WorkloadSummary][Standard performance workload]]
   * Repository size:  from 2 to 20 million artifacts (see [[#DataShape][data shape details]])
   * Additional details on jazz.net:   [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/501030][501030: Oracle 2m tests (SoftLayer)]]
   * Number of contributions in a global configuration:   250

#ArtifactCountDB2
---+++ Impact of total artifact count on DB2

As the number of artifacts grows, you must increase the size of the DB2 buffer pool for RM (as [[#DatabaseMemoryDB2][described in an earlier section]]).  For a system with sufficient memory, there is a slight increase in response times across most use cases as the repository grows, but performance is not that sensitive to repository size. The majority of the operations remain under 1 second for a repository with 10 million artifacts, and the difference in times between a 2 million and a 10 million artifact repository is typically less than 500 milliseconds.   In fact, the standard deviation is high enough that we can't conclude with certainty that there is a statistically significant degradation.

The following chart compares the response times of common browse and edit operations across different repository sizes. These operations use a GC with 250 contributions. The results have been filtered to include only those operations that take more than 1 second. 

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/RepoSizeImpactDB2.png" alt="RepoSizeImpactDB2.png"> 

The more complex operations are not very sensitive to repository size.   The exception is the operations involved in delivering a large change set;  these increased by 15% between the 2m and the 10m repository.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/RepoSizeImpactTransDB2.png" alt="RepoSizeImpactDB2Trans.png"> 

There is a slight increase in CPU usage on the RM server as the repository size grows (51% at 2m and 53% at 10m).   Disk utilization on the DB2 server increases moderately when moving from 2 million to 10 million:

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/RepoSizeDiskDB2.png" alt="RepoSizeDiskDB2.png"> 

About these tests:
   * [[#TestEnv4][Test environment 4]]
   * [[#Workload][Standard performance workload]]
   * Repository size:  10 million artifacts (see [[#DataShape][data shape details]])
   * Additional details on jazz.net:   [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/501472][Test/Benchmark DNG7RC5: DB2 on softlayer systems - 2Mi Small GC baseline, and Summary & comparisons of 10Mi vs 5Mi vs 2Mi rundata with fixed transactions]]

#UserLoad
---++ Impact of user load

#UserLoad20m
---+++ User load impact:  20 million artifacts, 2500 contribution GC

This section looks at what happens when the user load is increased from 500 users to 1000 users. The test was executed in a Oracle repository with 20 million artifacts, using a global configuration with 2500 contributions. The user load started at 500, and increased to 1000 in increments of 100. Each stage runs for 1 hour.

Summary of results:

   * Response times are stable up to (and including) 700 users.   At 800 users, response times start to degrade - so 700 users is the limit for this test.
      * *7.0.1 update*:  Response times are stable up to 1000 users in 7.0.1.  
   * CPU usage on the DNG server increases as user load increases, but CPU was not a bottleneck.
   * Disk usage on the Oracle server is close to 90%, but this is not a bottleneck.
   * Response times increase at 800 users due to a bottleneck in the DOORS Next server code.
 
The chart below shows the response times for common browse and edit operations, and shows how those response times vary as the user load increases.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ResponseTimes20m2500GC.png" alt="ResponseTimes20m2500GC.png"> 

Here's a chart for just one use case (saving a comment).

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/SaveCommentUsers.png" alt="SaveCommentUsers.png"> 

   About this test:
   * Number of simulated users:   500-1000
   * [[#TestEnv2][Test environment 2]]
   * [[#WorkloadSummary][Standard performance workload]]
   * Oracle MEMORY_TARGET:   48g
   * Number of requirements:  18,756,265
   * Number of versions: 118,928,959
   * Number of contributions in a global configuration:  2500
   * jazz.net work item:  [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/502062][Run at higher user loads (20m Oracle)]]

---+++ User load impact:  10 million artifacts, 1250 contribution GC

This section looks at what happens when the user load is increased from 500 users to 900 users.   The test was executed in a Oracle repository with 10 million artifacts, using a global configuration with 1250 contributions.   The user load started at 500, and increased to 900 in increments of 100.   Each stage runs for 1 hour.

Summary of results:
   * There is a slight increase in response time when comparing 900 users to 500 users, but the increase is small (.5 seconds or less).   The system could handle 900 users without an issue.
   * There is one scenario that slows down significantly as user load increases:   Creating a Word document from a module.    This is 30 seconds slower at 900 users.
   * Disk usage on the Oracle server increases as user load increases
   * CPU usage on the DNG server increases as user load increases

The next chart shows how DNG CPU and Oracle disk usage increase with user load.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/OSUsage.png" alt="OSUsage.png"> 

Here are the response times for common browse and edit operations for each user stage.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ResponseTimesVsUserLoad.png" alt="ResponseTimesVsUserLoad.png"> 

Here are the response times for the more complex transactions (creating streams or baselines, delivering change sets, or creating reports from modules).    The [[#CreateWordDoc]["Create Word document from module"]] scenario is the only one that is strongly affected by the user load.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ResponseTimesVsUserLoadTrans.png" alt="ResponseTimesVsUserLoadTrans.png"> 

About this test:
   * Number of simulated users:   500-900
   * [[#TestEnv2][Test environment 2]]
   * [[#WorkloadSummary][Standard performance workload]]
   * Oracle MEMORY_TARGET:   48g
   * Number of requirements:  9,447,009
   * Number of versions: 58,106,678
   * Number of contributions in a global configuration:   1250
   * jazz.net work item:  [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/501278][Run at higher user loads (10m Oracle)]]

#ProjectSize
---++ Impact of project area size

This section looks at the performance impact of the overall size of the project area.   We compared response times for the standard 500 user workload in two different cases:
   * Selecting random components in the small (200,000 artifacts) and medium (1 million artifacts) projects
   * Selecting random components in the large (3.8 million artifacts) project.

The response times for both cases are very similar;   operations are not very sensitive to project area size.  This makes sense because the SQL queries involved in these use cases select by configuration, so the overall size of the project area will not impact the way in which the database executes the query.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ProjectSizeImpact.png" alt="ProjectSizeImpact.png"> 

About this test:

   * [[#TestEnv2][Test environment 2]]
   * [[#WorkloadSummary][Standard performance workload]]
   * Repository size: 10 million artifacts (see [[#DataShape][data shape details]])
   * Additional details on jazz.net:   [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/501309][Impact of project area size (10m Oracle)]]
   * Number of contributions in a global configuration:   2500, with two different global configurations.   One GC only opens components in small and medium projects, and the other only opens components in the large project.


---++ Tested hardware and software configurations

#TestEnv1
---+++ Environment 1

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="center" disableallsort="on" tablewidth="90%" }%
| *Role* | *Server* | *Number of machines* | *Machine type* | *Processor* | *Total processors* | *Memory* | *Storage* | *Network interface* | *OS and version* |
| Proxy Server | IBM HTTP Server and !WebSphere Plugin | 1 | IBM System x3550 M3 | 2 x Intel Xeon X5667 3.07 GHz (quad-core) | 8 | 16 GB | RAID 5 – 279GB SAS Disk x 4 | Gigabit Ethernet | Red Hat Enterprise Linux Server 7 (Maipo) |
| RM server | !Embedded !WebSphere Liberty | 1 | IBM System x3550 M4 | 2 x Intel Xeon E5-2640 2.5GHz (six-core) | 24 | 32 GB | RAID 5 – 279GB SAS Disk x 4 | Gigabit Ethernet | Red Hat Enterprise Linux Server 7 (Maipo) |
| Jazz Team Server and Global Configuration Server | !Embedded !WebSphere Liberty | 1 | IBM System x3550 M4 | 2 x Intel Xeon E5-2640 2.5GHz (six-core) | 24 | 32 GB | RAID 5 – 279GB SAS Disk x 4 | Gigabit Ethernet | Red Hat Enterprise Linux Server 7 (Maipo) |
| Database | Oracle 19c | 1 | IBM System SR650 | 2 x Xeon Silver 4114 10C 2.2GHz  (ten-core) | 40 | 64 GB | RAID 10 – 900GB SAS Disk x 16 | Gigabit Ethernet | Red Hat Enterprise Linux Server 7 (Maipo) |
<br>

Storage details are below.  See [[DiskBenchmarking][Disk benchmarking]] for more details on I/O benchmarking.

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="left" disableallsort="on" tablewidth="35%" }%
|RAID controller |  Lenovo RAID 930-16i 4GB flash | 
| RAID mode | RAID10 |
| RAM (G) | 64 | 
| Total disks | 14 | 
| Spans | 7 | 
| Disks per span | 2 | 
| Drive speed (Gbps) | 12 | 
| Strip size (Kbyte) | 256 | 
| Logical sector size (bytes) | 512 | 
| Caching mode | Write-back | 
| Disk type| 900G Lenovo !ST900MP0146  | 
| Spin rate (rpm) | 15000 |
| Calibrate_io - IOPS| 6896|
| Calibrate_io - Mbps| 449|
| SLOB IOPS| 22581|
| Sysbench random r/w | |
|   * IOPS | 989|
|   * Read !MiB/s | 4.1|
|   * Write !MiB/s | 2.7|
| Sysbench sequential read| |
|   * IOPS| 15280|
|   * Read !MiB/s| 239|
<br>

Additional system settings: 
   * -Xmx20G -Xms20G -Xmn5G -Xcompressedrefs -Xgc:preferredHeapBase=0x100000000 -XX:MaxDirectMemorySize=1G
   * -Dcom.ibm.rdm.configcache.expiration=5040
   * -Dcom.ibm.rdm.configcache.size=5000
   * In rm/admin and gc/admin:
      * Maximum number of contributions per composite:  5000
      * JDBC connection pool:   400
      * RDB Mediator pool:  400
      * Maximum outgoing HTTP connections per destination: 400
      * Maximum outgoing HTTP connections total: 500
   * In rm/admin:
      * View query threadpool size override:  500


#TestEnv2
---+++ Environment 2

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="center" disableallsort="on" tablewidth="90%" }%
| *Role* | *Server* | *Number of machines* | *Machine type* | *Total processors* | *Memory* | *Storage* | *Network interface* | *OS and version* |
| Proxy Server | IBM HTTP Server and !WebSphere Plugin | 1 |Virtual machine - 4 2Ghz CPUs | 4 | 16 GB | Virtual | Gigabit Ethernet | Redhat EL 7.0-64 |
| RM server | !Embedded !WebSphere Liberty | 1 | Virtual machine - 16 2Ghz CPUs) | 32 | 32 GB | Virtual | Gigabit Ethernet | Redhat EL 7.0-64 |
| Jazz Team Server and Global Configuration Server | !Embedded !WebSphere Liberty | 1 | Virtual machine - 16 2Ghz CPUs) | 32 | 32 GB | Virtual | Gigabit Ethernet | Redhat EL 7.0-64 |
| Database | Oracle 19c | 1 | 2x 2.6GHz Intel Xeon-Broadwell (E5-2690-V4) | 56 | 128 GB | RAID 10 –  600G Seagate !ST3600057SS x 32 | Gigabit Ethernet | Redhat EL 7.6-64 |
<br>

Storage details are below.  See [[DiskBenchmarking][Disk benchmarking]] for more details on I/O benchmarking.

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="left" disableallsort="on" tablewidth="35%" }%
|RAID controller |  AVAGO !MegaRAID SAS 9361-8i  | 
| RAID mode | RAID10 |
| RAM (G) | 128 | 
| Total disks | 32 | 
| Spans | 8 | 
| Disks per span | 4 | 
| Drive speed (Gbps) | 6 | 
| Strip size (Kbyte) | 64 | 
| Logical sector size (bytes) | 512 | 
| Physical sector size (bytes) | 512 | 
| Caching mode | Write-back | 
| Disk type| 600G Seagate !ST3600057SS  | 
| Spin rate (rpm) | 15000 |
| Calibrate_io - IOPS| 14087|
| Calibrate_io - Mbps| 1452|
| SLOB IOPS| 16119|
| Sysbench random r/w | |
|    * IOPS| 1064|
|    * Read !MiB/s| 4.4|
|    * Write !MiB/s| 2.9|
| Sysbench sequential read| |
|    * IOPS| 30431|
|    * Read !MiB/s| 475|
<br>

Additional system settings: 
   * -Xmx20G -Xms20G -Xmn5G -Xcompressedrefs -Xgc:preferredHeapBase=0x100000000 -XX:MaxDirectMemorySize=1G
   * -Dcom.ibm.rdm.configcache.expiration=5040
   * -Dcom.ibm.rdm.configcache.size=5000
   * In rm/admin and gc/admin:
      * Maximum number of contributions per composite:  5000
      * JDBC connection pool:   400
      * RDB Mediator pool:  400
      * Maximum outgoing HTTP connections per destination: 400
      * Maximum outgoing HTTP connections total: 500
   * In rm/admin:
      * View query threadpool size override:  500

#TestEnv3
---+++ Environment 3

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="center" disableallsort="on" tablewidth="90%" }%
| *Role* | *Server* | *Number of machines* | *Machine type* | *Processor* | *Total processors* | *Memory* | *Storage* | *Network interface* | *OS and version* |
| Proxy Server | IBM HTTP Server and !WebSphere Plugin | 1 | IBM System x3550 M3 | 2 x Intel Xeon X5667 3.07 GHz (quad-core) | 8 | 16 GB | RAID 5 – 279GB SAS Disk x 4 | Gigabit Ethernet | Red Hat Enterprise Linux Server 7 (Maipo) |
| RM server | !Embedded !WebSphere Liberty | 1 | IBM System x3550 M4 | 2 x Intel Xeon E5-2640 2.5GHz (six-core) | 24 | 32 GB | RAID 5 – 279GB SAS Disk x 4 | Gigabit Ethernet | Red Hat Enterprise Linux Server 7 (Maipo) |
| Jazz Team Server and Global Configuration Server | !Embedded !WebSphere Liberty | 1 | IBM System x3550 M4 | 2 x Intel Xeon E5-2640 2.5GHz (six-core) | 24 | 32 GB | RAID 5 – 279GB SAS Disk x 4 | Gigabit Ethernet | Red Hat Enterprise Linux Server 7 (Maipo) |
| Database | Oracle 19c | 1 | IBM System SR650 | 2 x Xeon Silver 4114 10C 2.2GHz  (ten-core) | 40 | 64 GB | RAID 10 – 900GB SAS Disk x 16 | Gigabit Ethernet | Red Hat Enterprise Linux Server 7 (Maipo) |
<br>

Storage details are below.  See [[DiskBenchmarking][Disk benchmarking]] for more details on I/O benchmarking.

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="left" disableallsort="on" tablewidth="35%" }%
|RAID controller |  Lenovo RAID 930-16i 4GB flash | 
| RAID mode | RAID10 |
| RAM (G) | 64 | 
| Total disks | 14 | 
| Spans | 7 | 
| Disks per span | 2 | 
| Drive speed (Gbps) | 12 | 
| Strip size (Kbyte) | 256 | 
| Logical sector size (bytes) | 512 | 
| Physical sector size (bytes) | 4096 | 
| Caching mode | Write-back | 
| Disk type| 900G Lenovo !ST900MP0146  | 
| Spin rate (rpm) | 15000 |
| Calibrate_io - IOPS| 15452|
| Calibrate_io - Mbps| 829|
| SLOB IOPS| 24768|
| Sysbench random r/w | |
|   * IOPS | 969|
|   * Read !MiB/s | 4.9|
|   * Write !MiB/s | 2.7|
| Sysbench sequential read| |
|   * IOPS| 16828|
|   * Read !MiB/s| 263|
<br>

Additional system settings: 
   * -Xmx24g -Xmn4000m -Xcompressedrefs -Xgc:preferredHeapBase=0x100000000 -XX:MaxDirectMemorySize=1G
   * -Dcom.ibm.rdm.configcache.expiration=5040
   * -Dcom.ibm.rdm.configcache.size=2000


#TestEnv4
---+++ Environment 4

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="center" disableallsort="on" tablewidth="90%" }%
| *Role* | *Server* | *Number of machines* | *Machine type* | *Total processors* | *Memory* | *Storage* | *Network interface* | *OS and version* |
| Proxy Server | IBM HTTP Server and !WebSphere Plugin | 1 |Virtual machine - 4 2Ghz CPUs | 4 | 16 GB | Virtual | Gigabit Ethernet | Redhat EL 7.0-64 |
| RM server | !Embedded !WebSphere Liberty | 1 | Virtual machine - 16 2Ghz CPUs) | 16 | 32 GB | Virtual | Gigabit Ethernet | Redhat EL 7.0-64 |
| Jazz Team Server and Global Configuration Server | !Embedded !WebSphere Liberty | 1 | Virtual machine - 16 2Ghz CPUs) | 16 | 32 GB | Virtual | Gigabit Ethernet | Redhat EL 7.0-64 |
| Database | DB2 11.1 | 1 | 2x 2.6GHz Intel Xeon-Broadwell (E5-2690-V4) | 56 | 128 GB | RAID 10 –  600G Seagate !ST3600057SS x 32 | Gigabit Ethernet | Redhat EL 7.6-64 |
<br>

Storage details are below.  See [[DiskBenchmarking][Disk benchmarking]] for more details on I/O benchmarking.

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="left" disableallsort="on" tablewidth="35%" }%
|RAID controller |  AVAGO !MegaRAID SAS 9361-8i  | 
| RAID mode | RAID10 |
| RAM (G) | 128 | 
| Total disks | 32 | 
| Spans | 8 | 
| Disks per span | 4 | 
| Drive speed (Gbps) | 6 | 
| Strip size (Kbyte) | 64 | 
| Logical sector size (bytes) | 512 | 
| Physical sector size (bytes) | 512 | 
| Caching mode | Write-back | 
| Disk type| 600G Seagate !ST3600057SS  | 
| Spin rate (rpm) | 15000 |
| Sysbench random r/w | |
|    * IOPS| 1064|
|    * Read !MiB/s| 4.4|
|    * Write !MiB/s| 2.9|
| Sysbench sequential read| |
|    * IOPS| 30431|
|    * Read !MiB/s| 475|
<br>

We used the following configuration settings for the IBM HTTP Server:

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="right" disableallsort="on" tablewidth="25%" }%
| !ThreadLimit | 100 |
|  !ServerLimit |  100 |
|  !StartServers |  2 |
|  !MaxClients  | 2500 |
|  !MinSpareThreads |  250 |
|  !MaxSpareThreads | 2500 |
|  !ThreadsPerChild |  100 |
|  !MaxRequestsPerChild | 0 |
| Plugin: !ServerIOTimeout | 9000 |
<br>

The JVM settings were:
   * JTS:   -Xmx16G  -Xms16G -Xmn4G
   * RM:  -Xmx24G -Xms24G -Xmn6G
   * JDBC connection pool:   400
   * RDB Mediator pool:  400

#WorkloadSummary
---++ Performance workloads

IBM® Rational® Performance Tester was used to simulate user activities. A Rational Performance Tester script was created for each use case. The scripts are organized by pages; each page represents a user action. Users were distributed into many user groups and each user group repeatedly runs one use case. 

The tests simulate 500 users, ramping up the users at the rate of 1 every 2 seconds.    There is a settle period of 15 minutes, and then the tests collect data for 2 hours.   The tests use an average think time of 30 seconds (this is the delay between the execution of operations).     For 500 users, this results in an overall load of 15-20 logical operations per second (which is roughly 200 HTTP requests per second).

This table shows the use cases and the number of users who were repeatedly running each script:

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="left" disableallsort="on" tablewidth="60%" }%
| *Use case* | * Description * | * % Of Users * | * Number of users * |
| Scroll Module | Open a module and scroll through the contents | 16% | 80 users |
| Hover and edit | Hover over an artifact and then change its title | 9% |  45 users |
| Upload file | Create a new artifact by uploading a 4MB file | 4% | 20 users |
| Add a comment  | Open an artifact in a module and add a comment| 4% | 20 users |
| Copy/Paste 25  | Perform bulk copy, paste and delete of 25 artifacts | 4% | 20 users |
| Create stream | Open a component and create a new stream | n/a | 1 user |
| Create baseline | Open a component and create a new stream | 2% | 10 users |
| Deliver Small Changeset in Module | Create 1 new artifact, 3 artifact edits, and 6 comments in a module and deliver changeset | 16% | 80 users |
| Discard Small Changeset in Module | Create 1 new artifact, 3 artifact edits, and 6 comments in a module and discard changeset | 9% | 45 users |
| Deliver Large Changeset in Module | Create 5 new artifact, 15 artifact edits, and 30 comments in a module and deliver changeset  | 4% | 20 users |
| Discard Large Changeset in Module | Create 5 new artifact, 15 artifact edits, and 30 comments in a module and discard changeset  | 4% | 20 users |
| Display Module History | Display the revisions and audit history of a module | 4% | 20 users |
| Create Microsoft Word Document | Create a Microsoft Word document from a module | 2% | 10 users |
| View DNG Cross Component Links  | View query with links spanning multiple DNG components in the current GC configuration  | 9% | 45 users |
| Hover Over Cross Component Link | Hover over a cross component link | 4% | 29 users |
| Link DNG Cross Component Artifacts | Create a new artifact link to a different component in the current GC configuration  | 9% | 45 users |
<br>

Each of these use cases starts by logging in and navigating to the Global Configuration application and selecting a random GC of the appropriate size.   Next, the tests expand the nodes in the GC hierarchy and select a random stream.   Next, the tests open the list of modules and select a random module.   After this point, the tests execute the operations for the given use case.

#WorkloadRates
---+++ Rate of execution

When the workload reaches its peak of 500 users, it executes 16 operations per second (which is 163 HTTP transactions per second).    The breakdown by use case (executions per hour) is shown in the table below.

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="right" disableallsort="on" tablewidth="40%" }%
|	Operation	|	Per hour	|
|	Browse components (GC)	|	229	|
|	Open dashboard (GC)	|	228	|
|	Expand 2nd level GC	|	224	|
|	Expand 3rd level GC	|	220	|
|	Expand 4th level GC	|	220	|
|	Open a component in the GC app	|	227	|
|	Open a GC stream	|	226	|
|	Open a random leaf node in a GC	|	216	|
|	Search for a global configuration	|	228	|
|	Initialization after login	|	229	|
|	Login to GC app	|	230	|
|	Login - Jazz Authorization Server	|	230	|
|	Select a random artifact	|	3152	|
|	Add links columns to a module view	|	2149	|
|	Select "Create" in the Create Change set dialog	|	437	|
|	Select "Create change set…" in the configuration menu	|	439	|
|	Open the configuration menu	|	62	|
|	Collapse a section in a module	|	727	|
|	Copy/paste for multiple artifacts in a module	|	364	|
|	Prepare for a paste of multiple artifacts	|	364	|
|	Create a baseline	|	62	|
|	Select "OK" in the Create Links dialog	|	586	|
|	Finish delivering a change set that includes 10 links	|	66	|
|	Open the configuration menu	|	68	|
|	Select a random artifact in a module	|	598	|
|	Select a random component 	|	590	|
|	Select "Deliver Change Set" for a change set with 10 links	|	68	|
|	Select the "Artifact links" tab for an artifact	|	600	|
|	Select the "Links" icon 	|	603	|
|	Open a random module	|	587	|
|	Show a list of modules	|	589	|
|	Open the "Artifact Comments" tab	|	375	|
|	Save a comment	|	375	|
|	Select a random artifact in a module	|	375	|
|	Create a stream	|	2	|
|	Open the configuration menu	|	2	|
|	Create a Word document from a module	|	273	|
|	Select "Done" after editing an artifact	|	835	|
|	Select "Edit" to modify an artifact	|	840	|
|	Hover over an artifact in a module view	|	835	|
|	Hover over a link in a module view	|	528	|
|	Finish delivering a change set that includes 50 changes	|	10	|
|	Open the configuration menu	|	10	|
|	Select "Deliver Change Set" for a change set with 50 changes	|	10	|
|	Insert an artifact into a module 	|	50	|
|	Click away from an artifact in a view, to save it	|	237	|
|	Open the "Artifact Comments" tab	|	377	|
|	Save a comment	|	374	|
|	Click on the "Selected Artifact" tab after selecting an artifact in a view	|	379	|
|	Click on "Edit" to edit an artifact	|	234	|
|	Insert an artifact into a module 	|	50	|
|	Click on "Discard a change set" to discard 50 changes	|	10	|
|	Click on "Discard a change set" to discard 10 changes	|	150	|
|	Open a module	|	2739	|
|	Open a module (10,000 artifacts)	|	11	|
|	Internal - find module size	|	833	|
|	Open a module (1500 artifacts)	|	303	|
|	Open a module (500 artifacts)	|	4122	|
|	Scroll through the pages in a module	|	4015	|
|	Select "Discard change set…" in the configuration menu	|	160	|
|	Select the "Artifacts" tab	|	62	|
|	Show module history	|	549	|
|	Show a list of modules	|	6668	|
|	Finish delivering a change set that includes 10 changes	|	224	|
|	Open the configuration menu	|	224	|
|	Select "Deliver Change Set" for a change set with 10 changes	|	224	|
|	Insert an artifact into a module 	|	398	|
|	Click away from an artifact in a view, to save it	|	1289	|
|	Open the "Artifact Comments" tab	|	2621	|
|	Save a comment	|	2627	|
|	Click on the "Selected Artifact" tab after selecting an artifact in a view	|	2619	|
|	Click on "Edit" to edit an artifact	|	1287	|
|	Insert an artifact into a module 	|	396	|
|	Login to RM app	|	230	|
|	Select a file to be uploaded	|	376	|
|	Select "Requirement" as the artifact type to be uploaded	|	378	|
|	Select the target folder for an upload	|	378	|
|	Select the "Create->Upload Artofact…" menu	|	379	|
|	Select "Show artifacts" to open a component	|	378	|
|	Select "OK" in the Upload File dialog	|	379	|
<br>

#ScenarioDetails
---+++ Scenario details

This section provides more detail on the automation scenarios, to provide you with more context when you are interpreting the performance results.

#InitialOperations
---++++ Initial operations:   selecting and opening a module

Most scenarios start by navigating into a global configuration and selecting a DNG contribution.   Then, we display a list of modules.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ShowModules.png" alt="ShowModules.png"> 

Next, we select a random module to open.   Test modules come in three sizes:
   * 500 artifacts
   * 1500 artifacts
   * 10000 artifacts

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/OpenModule.png" alt="OpenModule.png"> 

#AddingLinkColumns
---++++ Adding link columns to a view

When we simulate adding link columns to a view, we start by opening a module.   We bring up the "Change Column Display Settings" dialog and select the "Link From" and "Link To" link types.   Then, we select "OK".    

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/AddLinkColumns_1.png" alt="AddLinkColumns_1.png"> 
<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/AddLinkColumns_2.png" alt="AddLinkColumns_2.png"> 

We measure the time for the view to refresh and display links in the two link columns (see below):
<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/AddLinkColumns_3.png" alt="AddLinkColumns_3.png"> 

---++++ Creating links

The "create links" scenario creates a change set, and then adds 10 links to random artifacts in a module.   Finally, the changes are delivered.

Here are the details for the steps.   The initial operations are [[#InitialOperations][described here]].    Once a module is opened, the next step is to select an artifact:

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/SelectArtifact.png" alt="SelectArtifact.png"> 

Then, select the "links" tab in the right-hand navigator:

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/SelectLinksTab.png" alt="SelectLinksTab.png"> 

Then, select the "Add links to Artifact" icon, and select the "Link To" link type:

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/SelectLinkTo.png" alt="SelectLinkTo.png"> 

This brings up the "Create Links" dialog.   Here, we select a target component:

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/SelectComponent.png" alt="SelectComponent.png"> 

Then, we select a random module in the selected component.   That displays a list of artifacts in the target module.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/SelectModuleLink.png" alt="SelectModuleLink.png"> 

Then, we select a random artifact and select "OK" to create the link.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ConfirmCreateLink.png" alt="ConfirmCreateLink.png"> 

The above sequence is repeated 10 times to create 10 links.    Then, we [[#ChangeSetOperations][deliver the change set]].

#ChangeSetOperations
---++++ Creating and delivering change sets

The simulations for creating or editing artifacts all operate within a change set.   This section describes the change set-specific parts of the scenario in more detail.

To create a change set, select the configuration menu and then select "Create change set".   Enter a name and select "Create".   We measure the time required for both the changet set to be created and for the UI to refresh.    So, change set creation times include the time required to reload a module in the context of the new change set.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/CreateChangeSet.png" alt="CreateChangeSet.png"> 

To deliver a change set, we start by selecting "Deliver Change Set..." from the configuration menu.    We measure the time required for the "Deliver Change Set" dialog to appear.   

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/SelectDeliver.png" alt="SelectDeliver.png"> 

Finally, we select the "Express" delivery option and then "Deliver".    We measure the time required for the deliver to complete (including the refresh of the UI).

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/FinishDeliver.png" alt="FinishDeliver.png"> 

---++++ Hovering over artifacts and links

The basic Hover scenario opens a module and hovers over an artifact.   We measure the time needed for the artifact to finish rendering into the popup window.   

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/HoverOverArtifact.png" alt="HoverOverArtifact.png"> 

Then, the module title is modified and the module is saved.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ModuleEdit.png" alt="ModuleEdit.png"> 
<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ModuleEditDone.png" alt="ModuleEditDone.png"> 

The "Hover over link" scenario first opens a module and adds "Linked to" and "Linked From" columns to the view, as [[#AddingLinkColumns][described here]].   Then, we hover over a link in one of the columns.   We measure the time required for the popup to completely render.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/HoverOverLink.png" alt="HoverOverLink.png"> 

---++++ Artifact creation and editing

There are two scenarios that focus on making and delivering changes.    The only difference between the two scenarios is the number of changes.   In the "Large deliver" scenario, we add 5 artifacts, edit 15 artifacts, and make 30 comments (for a total of 50 changes).   In the "Small deliver" scenario, we add 1 artifact, edit 3 artifacts, and make 6 comments (for a total of 10 changes).

The scenarios start out by [[#InitialOperations][opening a module]] and [[#ChangeSetOperations][creating a change set]].

To add an artifact, we bring up the artifact menu by clicking to the left of the artifact in the view, and then we select "Insert New Artifact" -> "After".   We then click away from the artifact to save it.   

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/AddArtifactAfter.png" alt="AddArtifactAfter.png"> 

To edit the artifact, we change the title in-line in the view and click away from the artifact to save it.

To add a comment, click on an artifact in the module and then click on the "Selected Artifact" tab.   

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/SelectArtifactComment.png" alt="SelectArtifactComment.png"> 
<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/SelectArtifactTab.png" alt="SelectArtifactTab.png"> 

Click on "Artifact comments" in the right-hand navigator.   

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ArtifactComments.png" alt="ArtifactComments.png"> 

Then, add a comment and select "OK" in the create comment dialog.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/CommentIcon.png" alt="CommentIcon.png"> 
<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/CreateCommentDialog.png" alt="CreateCommentDialog.png"> 

After making the required number of changes, we then [[#ChangeSetOperations][deliver the change set]].

We also have variations of these two scenarios where we just discard the change set rather than delivering it.

---++++ Module history

The Module History scenario starts by opening a random component, and then opening a module in that compoment.   Then, we bring up the action menu and select "Open History".

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/OpenHistory.png" alt="OpenHistory.png"> 

The scenario measures the time required for the following screen to display.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ModuleHistory.png" alt="ModuleHistory.png"> 

---++++ Copy/paste multiple artifacts

This scenario does a bulk copy/paste within a module.   Our modules are organized into a hiearchy of requirements, and in this scenario, we start by collapsing two of the "level 2" sections.   

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ModuleStructure.png" alt="ModuleStructure.png"> 

We then select the 2 collapsed sections and do a Copy.    This copies the selected artifact and its children (for a total of 16 artifacts).  Finally, we select the first collapsed artifact, click on the context menu, and select "Paste 16 artifacts->Before".

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/Paste16.png" alt="Paste16.png"> 

We measure the time it takes for the paste to complete (e.g. "Pasting artifacts..." goes away).   But note that this is an asynchronous operation that does not block the UI.   You can still do other things while the paste is in progress.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/Pasting.png" alt="Pasting.png"> 

---++++ Create a stream or a baseline

The Create Stream scenario starts by picking a random component and selecting "Show Modules".    After the list of modules displays, we select the "Create Stream..." menu from the configuration picker, enter a stream name, and select "Create".   We use the option "Switch to the new stream".     We measure the time required for the stream to be created and for the UI to refresh in the context of the new stream.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/CreateStream.png" alt="CreateStream.png"> 

For creating a baseline, we start by picking a random component and then selecting "Show Artifacts".   Then, from the configuration picker, we select "Create Baseline...".   We measure the time required for the baseline to be created and for the UI to refresh.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/CreateBaseline.png" alt="CreateBaseline.png"> 

#CreateWordDoc
---++++ Create a Microsoft Word document from a module

This scenario creates a Microsoft Word document from a module.    We start by picking a random component, and then selecting "Show Modules".   Then, we open a random module.    Next, select "Create and Print Microsoft Word Document..." from the action menu:

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/CreateWordDocMenu.png" alt="CreateWordDocMenu.png"> 

The scenario uses options to include comments and titles but not attributes.    Select "OK" to start creating the Word document.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/CreateWordDocStart.png" alt="CreateWordDocStart.png"> 

We measure the time from when you select "OK" in the previous screen to when the document creation finishes and you are prompted to open it.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/CreateWordDocFinish.png" alt="CreateWordDocFinish.png"> 

---++++ Scroll through the artifacts in a module

This scenario picks a random component and then selects "Show Modules".    We then pick a random module from the list and open it.

Then, we scroll through several pages.   We measure the time required for each scroll operation.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ScrollModule.png" alt="ScrollModule.png"> 

---++++ Upload a 4MB file

This scenario uploads a 4MB file into a new artifact.

We start by picking a random component and then selecting "Show Artifacts".

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ShowArtifacts.png" alt="ShowArtifacts.png"> 

Then, we select "Create->Upload Artifact..."

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/UploadArtifactMenu.png" alt="UploadArtifactMenu.png"> 

We select the file to upload ([[https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/93314-_DNG_4MB_FileforUpload.doc][DNG_4MB_FileForUpload.doc]]), select "Requirement" as the artifact type, and select the folder called "Generated Random Data" as the location for the uploaded document.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/StartUpload.png" alt="StartUpload.png"> 

We measure the time from selecting "OK" to the time when the "Upload File" dialog closes.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/UploadInProgress.png" alt="UploadInProgress.png"> 
<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/FolderAfterUpload.png" alt="FolderAfterUpload.png"> 

This is what the document looks like after uploading:

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/DocAfterUpload.png" alt="DocAfterUpload.png"> 

---++++ Add a comment to an artifact

This scenario picks a random module in a component, and opens an artifact in that module.   Then, it adds a comment to the artifact.

This scenario does not execute within a change set.


#DataShape
---++ Data shapes

The data shape for 7.0 is made up of a large number of standard-sized components, spread across 3 standard project area sizes.    When we want to increase the size of repository, we just add more of the standard projects.    In addition, the test environment is enabled for configuration management, and all operations are executed within the context of a global configuration.    We have 3 standard sizes for global configurations (small, medium and large).

---+++ Artifact counts

We have 3 different standard module sizes:
   * Small (200 artifacts)
   * Medium (1500 artifacts)
   * Large (10,000 artifacts)

We use these standard module sizes to create 3 different standard components.

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="right" dataalign="right" disableallsort="on" tablewidth="50%" }%
| * Artifact type* | * Small component * | * Medium component * | * Large component * |
| Number of large modules | 0  | 0 | 1 |
| Number of medium modules | 0  | 3 | 0 |
| Number of small modules | 20  | 0 | 0 |
| Total module artifacts | 4200  | 4500 | 10000 |
| Non-module artifacts | 100  | 100 | 100 |
<br>

We use these standard components to create 3 standard size projects (small, medium, and large).    The number of artifacts for these standard projects is summarized below.

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="right" dataalign="right" disableallsort="on" tablewidth="50%" }%
| * Artifact type* | * Small project* | * Medium project* | * Large project* |
| Large components | 1  | 5 | 20 |
| Medium components | 6  | 30 | 120 |
| Small components | 36  | 180 | 720 |
| Total components | 43  | 215 | 860 |
| Module artifacts | 181,000  | 905,000 | 3,620,000 |
| Non-module artifacts | 4,300  |  21,500 | 86,000 |
| Total artifacts | 185,300  |  926,500 | 3,706,000 |
| Large modules (10,000 artifacts) | 	1| 5 | 20 |
| Medium modules (1,500 artifacts)	| 18| 90 | 360|
| Small modules (200 artifacts)	| 720|  3,600 | 14,400 |
<br>

Finally, we combine these standard projects to create repositories of varying sizes.   Here is the summary of our test repositories up to 20,000,000 total artifacts.

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="right" dataalign="right" disableallsort="on" tablewidth="60%" }%
| Server Data	|	1m	|	2m	|	5m	|	10m	|	15m	|	20m	|
| Large Projects	|	0	|	0	|	0	|	1	|	1	|	1	|
| Medium Projects	|	0	|	1	|	2	|	2	|	6	|	10	|
| Small Projects	|	5	|	5	|	15	|	19	|	23	|	27	|
| Total Projects	|	5	|	6	|	17	|	22	|	30	|	38	|
| Module Artifacts	|	905,000	|	1,810,000	|	4,525,000	|	8,869,000	|	13,213,000	|	17,557,000	|
| Non-Module Artifacts	|	21,500	|	43,000	|	107,500	|	210,700	|	313,900	|	417,100	|
| Comments 	|	33,000	|	66,000	|	165,000	|	292,366	|	448,366	|	634,366	|
| RM Links	|	10,000	|	21,541	|	74,346 	|	178,076	|	187,851	|	279,104	|
| Large modules (10,000 artifacts)	|	5	|	10	|	25	|	49	|	73	|	97	|
| Medium modules (1,500 artifacts)	|	90	|	180	|	450	|	882	|	1,314	|	1,746	|
| Small modules (200 artifacts)	|	3,600	|	7,200	|	18,000	|	35,280	|	52,560	|	69,840	|
| Large Components	|	5	|	10	|	25	|	49	|	73	|	97	|
| Medium Components	|	30	|	60	|	150	|	294	|	438	|	582	|
| Small Components	|	180	|	360	|	900	|	1,764	|	2,628	|	3,492	|
| Total Components	|	215	|	430	|	1,075	|	2,107	|	3,139	|	4,171	|
| GC Components	|	51	|	102	|	256	|	511	|	767	|	1,023	|
| Total GC Baselines	|	2,557	|	5,115	|	12,788	|	25,575	|	38,363	|	51,150	|
| Total DNG Local Baselines	|	25,000	|	50,000	|	125,000	|	250,000	|	375,000	|	500,000	|
| Total Artifacts	|	1,023,782	|	1,934,201	|	4,835,488|	9,447,009	|	14,947,224|	18,756,265	|
| Total Versions |	6,142,698	|	12,321,207	|	30,844,163|	58,106,678	|	87,434,252|	118,928,959	|
<br>

---+++ Global configurations

We created global configurations (GCs) and added the DNG components as contributions.    We used 4 standard sizes:
   * A tiny GC with 10 contributions
   * A small GC with 250 contributions, consisting of 25 tiny GCs
   * A medium GC with 1250 contributions, consistent of 6 small GCs
   * A large GC with 2500 contributions, consistent of 2 medium GCs

In other words, the global configurations are organized into a hierarchy.   The figure below illustrates this for the small GC:

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/GCHierarchy.png" alt="GCHierarchy.png"> 

The large GC looks like this:

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/GCHierarchy2.png" alt="GCHierarchy2.png"> 

#OracleTuning
---++ Oracle tuning

This section addresses Oracle configuration parameters used during the performance testing.

The most critical tuning parameter is the amount of memory available to Oracle.   This is discussed in detail in the [[#DatabaseMemoryOracle][database memory requirements]] section.

The Oracle optimizer has several features that attempt to adjust execution plans if the optimizer finds that its cardinality estimates are inaccurate.    With ELM applications, however, variation in counts is normal.   For example, modules have different numbers of artifacts,   view queries return different matches, and the sizes of global configurations varies.    This can lead to SQL plan instability, where the optimizer is frequently doing hard parses to try and find better plans.    These hard parses are expensive, and can lead to view timeouts.    We recommend disabling Oracle's adaptive features.    They don't typically improve on the initial execution plans, and the cost of the hard parses is too high to incur repeatedly.

The performance tests described in this article were executing using these settings:
   * optimizer_adaptive_plans = false
   * optimizer_adaptive_reporting_only = false
   * optimizer_adaptive_statistics = false
   * _optimizer_use_feedback= false

Even with these settings, Oracle will still do occasional hard parses.   For example, after statistics are updated, Oracle will invalidate cursors over a 5 hour window.   When this happens, there may be a few view timeouts, especially for views using link columns in global configurations.    The timeouts will stop after a few minutes.

Other settings:
   * We used a redo log size of 6G.   This keeps the rate of log switching low (1-2 per hour).
   * parallel_degree_policy is set to "manual".   We do not want Oracle to use parallel execution plans so we need to override the default of "auto" and set to manual.   Parallel execution can cause performance issues with some ELM SQL.
   * *filesystemio_options* is set to "setall".   This allows Oracle to use asynchronous I/O, and avoids double-caching via the Linux page cache.
   * processes = 4480
   * open_cursors = 10000
   * session_cached_cursors = 1000

We also used automatic memory management, setting MEMORY_TARGET and MEMORY_MAX_TARGET to our desired memory values, while setting the other memory parameter to 0.    Most tests used a MEMORY_TARGET of 48g (except for the runs that were specifically looking at the impact of memory).
   * memory_target = 48000M
   * memory_max_target = 48000M
   * sga_max_size=0
   * sga_target=0
   * pga_aggregate_limit=0
   * pga_aggregate_target=0

We strongly recommend using Oracle Enterprise Edition, with the diagnostic+tuning pack.   This gives you features for analyzing database performance (like AWR reports) that are not available in the standard edition.    

Finally, it is not possible to list all of the Oracle optimizer settings and their impact on the ELM applications.  It is therefore assumed that if not specified explicitly above,other than the settings we have recommended, Oracle settings should be set to their defaults for production use.
We have seen <i>cursor_sharing=FORCE</i> being used for troubleshooting deviations in Oracle query performance.  It may be necessary at times for this to be used, but this setting will override key application optimizations and should be set to <i>EXACT</i> after any troubleshooting has been performed.

#Db2Tuning
---++ DB2 tuning

This section addresses DB2 configuration parameters used during the performance testing.

We used the following settings for the transaction logs:

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="right" disableallsort="on" tablewidth="40%" }%
|  Log file size (4KB)  | (LOGFILSIZ) = 65536  |
|  Number of primary log files  | (LOGPRIMARY) = 80  |
|  Number of secondary log files   | (LOGSECOND) = 100  |
<br>

We used the following memory settings for the RM database:


%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="right" disableallsort="on" tablewidth="30%" }%
| Package cache size (4KB)  | 32,184| 
| Sort heap thres for shared sorts (4KB)  | 214,615| 
| Sort list heap (4KB)  | 39,868| 
<br>

We also made these changes:
   * make db heap size automatic: db2 update db cfg  using DBHEAP 8192 automatic 
   * make application heap size automatic: db2 update db cfg  using APPLHEAPSZ 10000 automatic 

For test runs that were not explicitly investigating DB2 memory requirements, we used the following values for the RM database:
   * Buffer pool size: 52G  
   * RM Database_memory: 62G 

We also disabled automatic runstats.   This reduces the frequency of SQL parsing, which in turn improves response times and reduces disk utilization on the DB2 server.  While it is good to keep the DB2 statistics up to date, it is not necessary to do that as frequently as "AUTO_STMT_STATS" does it.    The downside of RUNSTATS is that it will invalidate the package cache, which then requires parsing of the SQL as the cache is repopulated.   
   * db2 update db cfg using AUTO_TBL_MAINT OFF
   * db2 update db cfg using AUTO_RUNSTATS OFF
   * db2 update db cfg using AUTO_STMT_STATS OFF

We enabled LOB inlining in two places.   [[https://www.ibm.com/docs/en/db2/11.5?topic=performance-inline-lobs-improve][LOB inlining]] allows DB2 to use the buffer pool for reading LOB data, rather than using direct reads (which always go to disk).

%TABLE{ tableborder="2" cellpadding="2" cellspacing="2" cellborder="2" headeralign="center" dataalign="right" disableallsort="on" tablewidth="40%" }%
| Schema | Table | Column | 
| REPOSITORY | ITEM_STATES | ITEM_VALUE |
| REPOSITORY | CONTENT_STORAGE|  CONTENT_BYTES |
<br>

For ITEM_VALUE, we used an inline length of 1200 bytes;   for content_bytes, we used an inline length of 4096 bytes.

<verbatim>
ALTER TABLE REPOSITORY.ITEM_STATES ALTER COLUMN item_value SET INLINE LENGTH 1200;
reorg TABLE REPOSITORY.ITEM_STATES LONGLOBDATA;

ALTER TABLE REPOSITORY.CONTENT_STORAGE ALTER COLUMN content_bytes SET INLINE LENGTH 4096;
reorg TABLE REPOSITORY.CONTENT_STORAGE LONGLOBDATA
</verbatim>

Finally, we recommend to apply the latest fixpack for DB2 v11.1, or set the following performance registry variable to the DB2 instance as DB2 administrator:
<verbatim>
 db2set DB2COMPOPT=OJSJ 
</verbatim>
, restart the DB2 instance to take effect. 

This is not required for DB2 v11.5.

#DngTuning
---++ DOORS Next server tuning - application

Server settings are listed in the sections describing the test environments, but a few of the settings are worth additional discussion.

The first setting controls the size of the configuration caches.   This is a JVM system setting (-Dcom.ibm.rdm.configcache.size=600) - the default is 600.   You should set this value to be the number of components that are going to be used by people over the course of the day.    In our test workloads, we open components randomly using a global configuration, so we set com.ibm.rdm.configcache.size to be greater than the number of contributions in our largest global configuration (2500).

If the cache is undersized, response times can be 5-10 seconds slower for some use cases at high user loads, as shown in the chart below.

Be aware that more memory is used as the cache size grows.   Each global configuration (including personal GCs) can require an additional 50M of memory, especially in those scenarios involving links.   If your JVM heap size is 24G or less, be cautious when increasing the cache size above 1000 (check your verbose GC logs for signs of trouble).    You may need a maximum heap size of 32G or larger to support cache sizes of 2500 or more.    

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/ConfigCache.png" alt="ConfigCache.png"> 

We also increased the cache expiration time:   -Dcom.ibm.rdm.configcache.expiration=5040.   By default, values expire from DNG caches after 10 minutes.   We didn't see a large performance impact from this parameter, since values are not immediately ejected when they expire.   But a longer expiration period (1 day or more) seems reasonable. 

[[https://www.ibm.com/support/pages/tuning-configuration-cache-ibm-engineering-requirements-management-doors-next-v7x][Tuning the configuration cache for IBM Engineering Requirements Management DOORS Next V7.X]] describes how to query for your active configurations in order to calculate the optimal configuration cache size.

By default, the number of contributions in a global configuration is limited to 1000.  We support larger global configurations on Oracle, so be sure to increase the setting.   This can be found in the rm/admin Advanced Properties UI (see below).    When you set this to a value greater than 1000, you'll see a warning message - you can ignore it.

<img src="https://jazz.net/wiki/pub/Deployment/RequirementsManagement70Performance/CompositeLimit.png" alt="CompositeLimit.png"> 

If you expect to have hundreds of active DNG users, you should consider raising the following values (all are Advanced Properties in rm/admin)
   * View query threadpool size override
   * JDBC connection pool
   * RDB Mediator pool
   * Maximum outgoing HTTP connections per destination
   * Maximum outgoing HTTP connections total

You can set these parameters to be the expected number of active users.

For some test runs, the response time for the scenario [[#CreateWordDoc]["Create Word document from module"]] was high (more than 1 minute).   This is in part because there is some throttling in place to limit the number of reports than can run in parallel.    The default is to limit the system to 3 simultaneous reports.   In our workload, we had 10 simulated users creating a Word document from a module, and so the throttling occasionally kicked in, increasing the response times.    The limit is controlled by the property "RDMPublishReportsRunners" in rm/admin Advanced Properties.

If there are Project Associations to various other servers, then note that the number of associations can add to the number of connections between the OSLC applications.  For Rhapsody Model Manager/Architecture Management project associations, we advise setting <i>EnableDmOSLCQueryPerformanceForSelectFilter</i> to *true*. <br>
If there are a significant number of <i>Dervies Architeture Elements</i>, or AM project associations, then also monitor the connection pools listed above and increase if you see any Queue Lengths in the Counters page.
</br>
See: [[https://jazz.net/wiki/bin/view/Deployment/HTTP-Connections-For-Server-To-Server-Calls][ELM - HTTP-Connections-For-Server-To-Server-Calls]] for further information on monitoring thread usage in real time.
</br>

---++ DOORS Next server tuning - operating system

As your system demands grow, you will need to increase the operating system kernel settings accordingly.  The Interactive Installation Guide describes the minimum set of values.

Failure to adequately monitor the system resources and the demand can lead to out of memory warnings from the web server layer, or out of threads at the Java Virtual Machine level.

Where we have encountered [[https://www.ibm.com/support/pages/insufficient-ulimit-u-nproc-value-causes-native-outofmemory-warning-message-be-issued][native out of memory issues]] within !WebSphere, we have increased the following settings in the Linux /etc/security/limits.conf file.  Note, if running a Linux service, use appropriate systemd files as articulated in [[https://access.redhat.com/solutions/1257953][How to set limits for services in RHEL and systemd]].

   * *      -  nofile  131072
   * *      -  nproc  131072 

Depending on how the system is setup, also check for other cgroup or kernel settings that influence thread allocation.  Some examples are below, but the location of where these are set will vary depending on whether you use systemd to initialize the JVM.

   * *      -  kernel.threads-max
   * *      -  kernel.pid_max
   * *      -  vm.max_map_count 




---++ Summary of revisions

3/11/2020:
   * Added section summarizing [[#DataShapeAndWorkload][data shape and workload changes]]
   * In progress - additional [[#ScenarioDetails][workload documentation]] and sanitizing of page names
3/12/2020
   * Added [[#DatabaseMemoryDb2][DB2 memory section]]
3/13/2020
   * Completed first pass through [[#ScenarioDetails][workload documentation]]
   * Added [[#UserLoad][section describing a staged run to 900 users]]
3/14/2020
   * Added [[#ProjectSize][How project area size impacts performance]]
3/15/2020
   * Edit pass
3/18/2020
   * Added section [[#ArtifactCountDB2][Impact of total artifact count on DB2]]
   * Added section [[#GlobalConfSizeDB2][Impact of GC size on DB2]]
3/19/2020
   * Added section [[#TenMillionDB2][Test results for a 10 million artifact repository (DB2)]]
   * Added section [[#Db2Tuning][describing DB2 configuration settings used in the tests]]
3/20/2020
   * Minor edits
3/23/2020
   * Added section [[#UserLoad20m][User load impact:  20 million artifacts, 2500 contribution GC]]
3/24/2020
   * Added section [[#DngTuning][DNG server tuning]]
3/27/2020
   * Updated [[#GlobalConfSizeDB2][the section on GC size and DB2]] to reflect improvements from tuning.
   * Updated [[#Db2Tuning][the DB2 tuning section]] to recommend disabling automatic runstats.
   * Updated [[#TenMillionDB2][10 million DB2 results]] to reflect improvements from tuning.   Increased the supported GC size to 1250 contributions for DB2.
3/30/2020
   * Fixed transaction names in tables and charts.  "Finish and deliver" transactions were incorrectly labeled as "Select and deliver".
   * Included a [[#WorkloadRates][summary of the rate of execution for various use cases]]
4/30/2020
   * Updated [[#UserLoad20m][user load tests]] to reflect 7.0.1 improvements (tested up to 1000 users)
   * Added discussion on the memory impact of increasing DNG cache sizes
8/24/2022
   * Added kernel settings considerations when scaling

---+++++!! Related topics: 
   * [[OracleSQL][Understanding the performance of SQL statements in Oracle]]
   * [[DiskBenchmarking][Disk benchmarking]]



---+++++!! External links:          

   * [[https://www.ibm.com][IBM]]

<sticky></div></sticky>
Attachments
Topic attachments
I	Attachment	Action	Size	Date	Who
doc	93314-_DNG_4MB_FileforUpload.doc	manage	4046.0 K	2020-03-13 - 15:28	VaughnRokosz
png	AddArtifactAfter.png	manage	86.7 K	2020-03-12 - 16:50	VaughnRokosz
png	AddLinkColumns_1.png	manage	3.8 K	2020-03-11 - 18:25	VaughnRokosz
png	AddLinkColumns_2.png	manage	13.1 K	2020-03-11 - 18:25	VaughnRokosz
png	AddLinkColumns_3.png	manage	11.3 K	2020-03-11 - 18:25	VaughnRokosz
png	ArtifactComments.png	manage	10.0 K	2020-03-12 - 16:51	VaughnRokosz
png	ArtifactCountImpact.png	manage	16.6 K	2020-03-12 - 15:02	VaughnRokosz
png	ArtifactCountImpactTrans.png	manage	19.5 K	2020-03-30 - 15:11	VaughnRokosz
png	BufferCacheExample.png	manage	10.6 K	2020-03-03 - 17:26	VaughnRokosz
png	CommentIcon.png	manage	5.4 K	2020-03-12 - 16:51	VaughnRokosz
png	CompositeLimit.png	manage	8.4 K	2020-03-24 - 19:02	VaughnRokosz
png	ConfigCache.png	manage	38.2 K	2020-03-24 - 18:48	VaughnRokosz
png	ConfirmCreateLink.png	manage	39.7 K	2020-03-12 - 13:22	VaughnRokosz
png	CreateBaseline.png	manage	7.3 K	2020-03-13 - 13:54	VaughnRokosz
png	CreateChangeSet.png	manage	7.4 K	2020-03-12 - 13:53	VaughnRokosz
png	CreateCommentDialog.png	manage	10.5 K	2020-03-12 - 16:51	VaughnRokosz
png	CreateStream.png	manage	7.3 K	2020-03-13 - 13:54	VaughnRokosz
png	CreateWordDocFinish.png	manage	6.9 K	2020-03-13 - 14:54	VaughnRokosz
png	CreateWordDocMenu.png	manage	22.9 K	2020-03-13 - 14:54	VaughnRokosz
png	CreateWordDocStart.png	manage	6.2 K	2020-03-13 - 14:54	VaughnRokosz
png	DB2Bufferpool.png	manage	12.3 K	2020-03-12 - 17:51	VaughnRokosz
png	DB2Memory.png	manage	7.9 K	2020-03-12 - 17:33	VaughnRokosz
png	DocAfterUpload.png	manage	90.1 K	2020-03-13 - 15:29	VaughnRokosz
png	FinishDeliver.png	manage	32.4 K	2020-03-12 - 13:53	VaughnRokosz
png	FolderAfterUpload.png	manage	16.0 K	2020-03-13 - 15:29	VaughnRokosz
png	GCCPUImpact.png	manage	8.4 K	2020-03-03 - 20:11	VaughnRokosz
png	GCHierarchy.png	manage	56.3 K	2020-02-27 - 15:53	VaughnRokosz
png	GCHierarchy2.png	manage	60.5 K	2020-02-27 - 16:05	VaughnRokosz
png	GCSizeImpact.png	manage	19.2 K	2020-03-15 - 13:48	VaughnRokosz
png	GCSizeImpactDB2.png	manage	61.9 K	2020-03-27 - 13:09	VaughnRokosz
png	GCSizeImpactTransDB2.png	manage	18.0 K	2020-03-30 - 15:05	VaughnRokosz
png	GCSizeImpactTransactions.png	manage	14.6 K	2020-03-15 - 13:50	VaughnRokosz
png	HoverOverArtifact.png	manage	71.9 K	2020-03-12 - 16:15	VaughnRokosz
png	HoverOverLink.png	manage	12.2 K	2020-03-12 - 16:15	VaughnRokosz
png	ModuleEdit.png	manage	13.3 K	2020-03-12 - 16:15	VaughnRokosz
png	ModuleEditDone.png	manage	18.3 K	2020-03-12 - 16:15	VaughnRokosz
png	ModuleHistory.png	manage	11.9 K	2020-03-12 - 22:10	VaughnRokosz
png	ModuleStructure.png	manage	36.2 K	2020-03-12 - 22:34	VaughnRokosz
png	OSUsage.png	manage	7.4 K	2020-03-13 - 19:02	VaughnRokosz
png	OpenHistory.png	manage	22.6 K	2020-03-12 - 22:09	VaughnRokosz
png	OpenModule.png	manage	26.5 K	2020-03-11 - 18:24	VaughnRokosz
png	OramemByTablespace.png	manage	9.8 K	2020-03-03 - 17:27	VaughnRokosz
png	PageRespVCache.png	manage	10.2 K	2020-03-11 - 19:36	VaughnRokosz
png	Paste16.png	manage	58.1 K	2020-03-12 - 22:34	VaughnRokosz
png	Pasting.png	manage	19.4 K	2020-03-12 - 22:34	VaughnRokosz
png	ProjectSizeImpact.png	manage	16.9 K	2020-03-15 - 13:57	VaughnRokosz
png	RepoSizeDiskDB2.png	manage	8.4 K	2020-03-18 - 19:47	VaughnRokosz
png	RepoSizeImpactDB2.png	manage	26.0 K	2020-03-18 - 19:47	VaughnRokosz
png	RepoSizeImpactTransDB2.png	manage	19.6 K	2020-03-30 - 15:24	VaughnRokosz
png	RepoSizes.png	manage	10.5 K	2020-03-03 - 17:27	VaughnRokosz
png	ResponseTimes20m2500GC.png	manage	41.7 K	2020-03-23 - 18:37	VaughnRokosz
png	ResponseTimesVsUserLoad.png	manage	70.4 K	2020-03-13 - 19:02	VaughnRokosz
png	ResponseTimesVsUserLoadTrans.png	manage	18.1 K	2020-03-30 - 16:06	VaughnRokosz
png	SaveCommentUsers.png	manage	5.3 K	2020-03-23 - 18:37	VaughnRokosz
png	ScrollModule.png	manage	16.7 K	2020-03-13 - 14:57	VaughnRokosz
png	SelectArtifact.png	manage	9.0 K	2020-03-12 - 13:22	VaughnRokosz
png	SelectArtifactComment.png	manage	7.4 K	2020-03-12 - 16:52	VaughnRokosz
png	SelectComponent.png	manage	14.0 K	2020-03-12 - 13:22	VaughnRokosz
png	SelectDeliver.png	manage	18.4 K	2020-03-12 - 13:53	VaughnRokosz
png	SelectLinkTo.png	manage	2.8 K	2020-03-12 - 13:22	VaughnRokosz
png	SelectLinksTab.png	manage	39.4 K	2020-03-12 - 13:22	VaughnRokosz
png	SelectModuleLink.png	manage	7.3 K	2020-03-12 - 13:23	VaughnRokosz
png	SelectedArtifactTab.png	manage	9.8 K	2020-03-12 - 16:52	VaughnRokosz
png	ShowArtifacts.png	manage	2.4 K	2020-03-13 - 15:28	VaughnRokosz
png	ShowModules.png	manage	22.4 K	2020-03-11 - 18:24	VaughnRokosz
png	StartUpload.png	manage	10.7 K	2020-03-13 - 15:29	VaughnRokosz
png	UploadArtifactMenu.png	manage	7.0 K	2020-03-13 - 15:29	VaughnRokosz
png	UploadInProgress.png	manage	19.1 K	2020-03-13 - 15:29	VaughnRokosz
png	awr_cache.png	manage	120.2 K	2020-03-03 - 17:26	VaughnRokosz
png	awr_io.png	manage	64.9 K	2020-03-03 - 17:26	VaughnRokosz
Copyright © by IBM and non-IBM contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.