r1 - 2018-04-05 - 10:39:03 - LeeByrnesYou are here: TWiki >  Deployment Web > DeploymentPlanningAndDesign > PerformanceDatasheetsAndSizingGuidelines > SizingAndTuningGuideForRationalDOORSNextGeneration604

Sizing and tuning guide for Rational DOORS Next Generation 6.0.4 (DRAFT)todo.png

Authors: Mahesh Gunbote, Lee Byrnes
Build basis: DNG 6.0.4 ifix 6.

Introduction

The 6.0.4 ifix 6 release of IBM® Rational® DOORS® Next Generation (RDNG) introduced several changes to the performance of the overall product architecture and the purpose of this set of testing was to update the sizing guide.

This 6.0.4 sizing guide focuses on the improvement of the scalability of a deployment (where scalability refers to the maximum throughput/user load which can be achieved for a given repository size without significant response degradation) and it's overall response times.

Standard Disclaimer

The information in this document is distributed AS IS. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk. Any pointers in this publication to external Web sites are provided for convenience only and do not in any manner serve as an endorsement of these Web sites. Any performance data contained in this document was determined in a controlled environment, and therefore, the results that may be obtained in other operating environments may vary significantly. Users of this document should verify the applicable data for their specific environment.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the number of other applications running in the customer environment, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

This testing was done as a way to compare and characterize the differences in performance between different configurations of the 6.0 product. The results shown here should thus be looked at as a comparison of the contrasting performance between different configurations, and not as an absolute benchmark of performance.

How hardware configurations impact scalability

We conducted a series of tests with different hardware configurations, to assess the impact of memory and disk drives on scalability. These tests cover the same configurations as in the 6.0 sizing guide, with the exception of the SSD tests which are no longer part of the test sequence. We carried out these tests by setting up a specific hardware configuration and repository size, and then slowly increasing the number of simulated users until we started to see increasing response times.

As a general rule, the system can support more users (and more artifacts) as the amount of RAM on the DNG server increases. The DNG server stores all the data in use in indices local to the RM server and will cache this index in memory if possible, which is why increasing the amount of RAM can improve performance. The amount of RAM available to cache the DNG index is roughly the total amount of RAM minus the maximum size of the JVM heap. If the disk size of the DNG index (in server/conf/rm/indices) is less than that number, then the DNG index will be cached in memory, which will allow for the fastest read access to DNG data. The DNG index will be accessed from the file system when there isn't enough memory to cache the index completely (but also on write operations).

A note on failure modes

What we see in all cases is that the response times slowly increase as the user load increases up to a point - and at that point, there is a sudden, sharp increase in response times. At the same time, the CPU utilization on the DNG server increases to 100%. This is the classic symptom of lock contention in Java. As the load increases, the simulated users start to ask for exclusive access to the same data, and this causes a queue to form. The bottleneck forms around access to the DNG index information. We have observed that the user load at which this happens is not consistent, and so there is a margin of error on the user estimates of +/- 50-100 users.

This failure mode is in part a consequence of the performance simulation itself, which issues a stream of requests against the server at a steady rate, without pause. Our simulated users never take breaks for coffee, or go to meetings, or go to lunch! This represents a worst case scenario, where the server can never catch up. In production usage, it would be common for bursts of operations to be followed by slow periods, during which the server can catch up.

Tuning recommendations for all configurations

JVM nursery sizing: 1/3 of max JVM heap

When allocating memory for the Java Virtual machine for the DNG server, allocate 1/3 of the total heap to the nursery (divide the value of -Xmx by 3, and use that for -Xmn).

The nursery is a section of JVM memory used to efficiently manage short-lived objects, such as those associated with handling an incoming HTTP request. As the number of concurrent DNG users grows, there is more demand on the nursery to process the incoming transactions. If the nursery is too small, memory for those objects will be copied into sections of JVM memory which have more management overhead. By allocating more memory to the nursery, you can improve performance at higher load levels by ensuring that short-lived objects stay in the nursery.

Tuning the configuration management cache

Some information about streams and baselines is cached in memory. The cache size can be changed to improve the performance for higher user loads (with the tradeoff being additional memory usage). You control the cache settings through JVM startup parameters (e.g. through the -D flag or by using custom JVM properties).

The cache stores information which DNG associates with each configuration (where a configuration is a stream, baseline, or change set). For best performance, set the cache size to the number of concurrent users. This supports the worst-case scenario where all active users are working in separate configurations. The default value for this cache is 200.

To set as a JVM startup parameter:

-Dcom.ibm.rdm.configcache.size=400

Assume that each active entry in the cache will consume 2M of RAM.

For additional tuning guidance for the configuration cache, refer to this technote Tuning the configuration cache for IBM Rational DOORS Next Generation

Keeping database statistics up-to-date

Information about streams, baselines, and changesets is stored in tables in the database. The DNG server will query these tables when users work in a DNG project which is enabled for configuration management. For best performance, be sure that the database statistics are kept up to date as the size of the repository grows. This is especially critical when you are first adopting configuration management. Some of the tables will start out empty, and then grow as users work with streams, baselines, and change sets. Keeping the database statistics updated ensures that the queries against those tables will be properly optimized.

Databases generally manage statistics automatically; for example, in a scheduled overnight operation. However, to ensure that the database is fully optimized, you can manually run the statistics as follows. DB2

DB2 REORGCHK UPDATE STATISTICS ON TABLE ALL 

Oracle

EXEC DBMS_STATS.GATHER_DATABASE_STATS

See also Slow server performance and CRRRW7556E on IBM Rational DOORS Next Generation version 6.0.x for further information on keeping databases tuned for configuration management, if you are experiencing performance issues.

Available TCP/IP ports

The default number of TCP/IP ports that are available on AIX and Linux operating systems is too low and must be increased. Windows operating systems have a higher default limit, but that limit might still be too low for large concurrent user loads. Use the following instructions to increase the port range.
  • On AIX and Linux systems, set the limit as follows: ulimit -n 65000
  • On Windows 2003 Server, follow these steps:
    • Open the registry.
    • Navigate to HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/Tcpip/Parameters and create a dWord named MaxUserPort. Set its value to 65000.
    • Restart the computer.
  • On Windows 2008 Server, follow the instructions in this Microsoft support article to change the dynamic port range. For the start port setting, use 2000. Set the number of ports to 63535.

Thread pool size

The WebSphere Application Server thread pool size must be increased from the default of 50 to 0.75 times the expected user load. For example, if you have 400 concurrent users, the thread pool maximum should be set to 300.

More suggestions

In a 3-server topology, where Jazz Team Server and the DNG server are on separate hardware, adjust two settings in the fronting.properties file for loads higher than 200 concurrent users.
  • Ensure that com.ibm.rdm.fronting.server.connection.route.max equals the number of users
  • Ensure that com.ibm.rdm.fronting.server.connection.max is twice the value of number of users
When you use a proxy or reverse proxy server, you might need to increase the maximum allowed connections on the proxy server, depending on the concurrent user load. For more information, see Tuning IBM HTTP Server to maximize the number of client connections to WebSphere Application Server .

Repositories with up to 500,000 artifacts

The chart below summarizes the results of testing with a repository of 500,000 artifacts, using a standard configuration management workload. Here we looked at the impact of the amount of RAM on the response times and maximum user limit. The system with 32G RAM performed the worst. Given that the 500K repository has a 34G index in DNG 6.0.2, and our 32G system has only 16G or less available for caching, this is not unexpected. Increasing the amount of RAM to 64G allows for most of the index to be cached in memory. Performance for both the 64G and 128G systems are better than the 32GB system and there is a notable improvement in user number support when compared to DNG 6.0.

The maximum user load for the 64G system was again higher than the 128Gb system (700 users vs. 600 for 128G). This is related to the inconsistencies discussed earlier regarding the exact point at which bottleneck around the DNG index forms. We therefore use 600 users as the limit for the 500,000 artifacts, to be conservative.

IMAGE FOR 500 GOES HERE

Note: in this graph, we have left out the data points collected when the system was overloaded. If you want to see all of the data points, please refer to the summary table.

The following table shows the various hardware configurations with RAM and heap sizes and the maximum number of concurrent users that are supported in each hardware configuration for the DNG 6.0.2 and DNG 6.0.4 releases. Again this is related to the inconsistencies discussed earlier regarding the exact point at which bottleneck around the DNG index forms.

RM server hardware and JVM configuration

Jazz Team Server

DB2 server

Number of users supported in DNG 6.0.2

Number of users supported in DNG 6.0.4

Change in user numbers

32 GB RAM with 16 GB JVM heap

32 GB RAM with 16 GB JVM heap

64 GB RAM

500

400

-100

64 GB RAM with 24 GB JVM heap

32 GB RAM with 16 GB JVM heap

64 GB RAM

500

500

0

128 GB RAM with 24 GB JVM heap

32 GB RAM with 16 GB JVM heap

64 GB RAM

500

500

0

Repositories with up to 1M artifacts

The chart below shows response times as a function of user load for a repository with 1 million artifacts, for a variety of hardware configurations. Here are the main observations:

  • More than 32G of RAM is needed to cache the index (68G) associated with 1M artifacts, so the response times and scale limits are lowest in 32G configurations.

IMAGE FOR 1000 GOES HERE

The following table shows the various hardware configurations with RAM, heap sizes and the maximum number of concurrent users that are supported in each hardware configuration for the DNG 6.0.2 and DNG 6.0.4 releases. Again this is related to the inconsistencies discussed earlier regarding the exact point at which bottleneck around the DNG index forms.

RM server hardware and JVM configuration

Jazz Team Server

DB2 server

Number of users supported in DNG 6.0

Number of users supported in DNG 6.0.2

Improvement in user numbers

HDD – 32 GB RAM with 16 GB JVM heap

32 GB RAM with 16 GB JVM heap

64 GB RAM

400

100

-300

HDD – 64 GB RAM with 24 GB JVM heap

32 GB RAM with 16 GB JVM heap

64 GB RAM

500

500

0

HDD – 128 GB RAM with 24 GB JVM heap

32 GB RAM with 16 GB JVM heap

64 GB RAM

500

500

0

Response times and CPU utilization - all configurations

The table below summarizes the overall average response time and the CPU utilization of the DNG server at different user levels, for the tested hardware configurations. The cells in red indicate that the system was past its limit.
The yellow column shows the test that is missing this release due to a hardware failure

OVERALL TABLE GOES HERE

Heading 2 (use sentence-style capitalization)

Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text Sub-Section text

Related topics: Deployment web home, Deployment web home

External links:

Additional contributors: TWikiUser, TWikiUser

Edit | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r1 | More topic actions
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.
Ideas, requests, problems regarding the Deployment wiki? Create a new task in the RTC Deployment wiki project