IBM Rational Software Architect Design Manager 3.0 performance characteristics and tuning guide

Claire Liu and Ya Ju Yan, IBM
Last updated: September 15, 2011
Build basis: Rational Design Manager 3.0

Table of contents

Overview
Performance test environment
Workload characterization summary
Performance results
Server resource utilization levels
Hardware sizing
Tomcat and WebSphere Application Server configurations
Storage requirements
Network connectivity
Artifact sizing guidelines
For more information
About the authors

Overview

Performance and scalability depend on several factors that include, but are not limited to, hardware setup and tuning, software tuning, operating system tuning, network traffic and latency, and end user usage pattern. We, the authors, created a sophisticated performance test environment to simulate a realistic customer workload and to validate performance and scalability requirements for Rational® Software Architect Design Manager. With this workload testing, we were able to support up to 50 simultaneous users in an enterprise environment with less than 2 seconds interactive transaction response time. The server performance data that we collected during these tests is contained in this document.

Performance test environment

We set up an enterprise environment with a single application server to generate a test workload. The test environment consists of an application server, a DB2® server, an LDAP server, and three client computers as shown in Figure 1. The tests were run with different numbers of users, starting with single user and increasing to 10, 20, and 50 users. All users were regularly performing different interactive transactions every 2 minutes within nine active project areas. The interactive transactions include searching model elements and diagrams, opening diagrams, posting comments, creating reviews, posting review comments, and so on. The number of model elements stored in the database can affect performance significantly. 37 model projects containing 1,012,707 model elements were imported into nine project areas. The total size of model projects is 215 MB. The database size grew 1.5 GB after importing all models and was increased to 4 GB. Therefore, all our tests were run against the 4 GB database to ascertain the behavior of the system under load, and to validate the maximum user load. We also tested the model import process before starting the test run to validate the import performance requirement.

Figure 1. Topology diagram of the performance test environment (created in Rational Software Architect using the Deployment modeling tools)

The following table (Table 1) details the configuration of the specific computers and software used in the above topology. All computers are connected to the same subnet.

Table 1. Hardware and software details of the performance test environment

Workload characterization summary

The workload is divided by the various user roles of the software development design phase. The defined roles, commenter (stakeholder), architect, and project lead, attempt to mimic simple real-world user tasks. Within each role, various user workflows are broken out as a percentage of overall work done, as shown in Table 2.

Table 2. Role and scenario details

Performance results

Transaction performance

Table 3 depicts the average response time of various key transactions under different user loads and the graph representation is shown in Figure 2. We use Rational Performance Tester to simulate multiple user activities. Overall, Rational Software Architect Design Manager server achieves less than 2 seconds average response time with up to 50 concurrent users. These results can be used as a starting point for performance and scalability expectations for the product. Response times can differ from our test results due to hardware and network configuration and web browser differences. The response time in Table 3 is an average response time for all pages. The average response time for some individual pages is shown in Table 4.

Table 3. Average Response time with different user loads

Figure 2. Response times with different user loads

Performance comparison for the web client and Rational Software Architect client

Table 4 shows the comparison of average response time between the web client and Rational Software Architect client. The test was performed manually with a single user, rather than running Rational Performance Tester scripts. Therefore, the test results are closer to the real user experience. As we can see, opening a diagram in the Rational Software Architect client is faster than in the web client. The average response time for searching model elements in the web client is shorter than in the Rational Software Architect client. Both the web client and Rational Software Architect client show a similar response time for posting one comment.

Table 4. Performance comparison between the web client and Rational Software Architect client

Import performance

We imported different sizes of models into five active project areas during the import performance testing. The artifacts used for import testing are shown in Table 5, which includes the size of the model project, the number of resources, and the number of model elements imported. Table 6 shows the duration of the model import process for each project area and the average import speed. The chart of import duration against different model sizes is shown in Figure 3. Note that all the model imports in the test run did not enable the Validate model contents, Reimport and overwrite all resources, and Create snapshot of the entire project after import completes options.

Table 5. Artifact sizing metrics

Table 6 . Model import performance results

Figure 3. Import duration with different sizes of models

The duration of the model import process depends on the physical size of the model project, model complexity, and the selected import options. In Table 7, shown below, the duration of the model import process is almost double when the Reimport and overwrite all resources option is selected. If this option is selected, the import process deletes all existing resources in the project area and reimports the model, which results in a longer import duration. The Validate model contents option also increases the duration of the import process because the validation is performed. The Create snapshot of the entire project after import completes option does not affect the import duration. However, the snapshot is taken after the import. The duration of creating a snapshot depends on the size of model project. For example, it takes 2 minutes and 50 seconds to create a snapshot for a 7.01 MB model.

Table 7. Model import and reimport performance results

Server resource utilization levels

Processor and memory utilization were monitored on the application server during the test runs. Table 8 and Figure 4 show the utilization data for different user loads when running the tests. The CPU utilizations for different user loads are no more than 40% and memory usages are less than 15%.

Table 8. Rational Software Architect Design Manager Server resource utilization

Figure 4. CPU and memory usage with different user loads

Memory, heap, and CPU usage of the Java process with a 10-user load during a 1-hour test run are shown in Figure 5, Figure 6, and Figure 7. Based on our configuration with 8 GB RAM and 6 GB heap size for a 64-bit installation, the Java process consumes less than 13% RAM in this run (Figure 5). In Figure 6, the line on the bottom indicates that the used heap size by the Java process after garbage collection is 300 MB. As our test environment has a dual-core processor, the CPU utilization is around 60% most of the time during the run based on the total 2-core CPU resources as shown in Figure 7. You might notice that the CPU usage is over 100% sometimes in Figure 7. It represents the multithreaded Java process is using more than one core.

Figure 5. Memory utilization of Java process

Figure 6. Heap utilization of the Java process

Figure 7. CPU utilization of the Java process

Figure 8. Memory utilization of an import and reimport process

Figure 8 shows the memory usage during an import and re-import process for a 2.1 MB project. The two short lines indicate the memory usage for the first import and the other two lines indicate the reimport process. As noted in the Import performance section, the reimport process took longer than the first import because a delete resource operation was performed before the reimport process. The memory usage drops after the import or reimport is complete.

Hardware sizing

The hardware and software configuration we used for performance and scalability testing demonstrated the ability to handle up to 50 concurrent users on a single application server configuration, along with separate database and LDAP servers. Customers might increase or reduce configurations according to enterprise scales. The minimal hardware requirement for Rational Software Architect Design Manager Server is a 2-core CPU with at least 2 GB RAM. The memory usage depends on the number of concurrent users, model size being imported, and the size of database. More memory improves performance. To improve performance, customers should host Rational Software Architect Design Manager Server on a 64-bit operating system.

Tomcat and WebSphere Application Server configurations

By default, Jazz Team Server is preconfigured with Tomcat, which is sufficient for evaluation and small-scale departmental deployment purposes. WebSphere Application Server is preferable for large-scale and enterprise deployments.

Tomcat includes a simple, file-based user realm for LDAP and non-LDAP setups. All applications are deployed on the same (virtual) host. You can have multiple Tomcat servers running on the same physical computer with different port numbers.

WebSphere Application Server provides flexibility for creating multiple profiles, and has administrative, monitoring, security, and high availability benefits. WebSphere Application Server also offers a user interface for installing, stopping, and restarting applications, configuring Java virtual machine properties, and LDAP support.

Additional configurations are required for Rational Software Architect Design Manager Server to handle large enterprise model imports or multiple concurrent users.

If Rational Software Architect Design Manager applications are deployed on Tomcat, the following configurations are optimal:

1. Make sure you have a 64-bit Java virtual machine (JVM) installed and the server startup and shutdown files pointing to the correct JVM installation path. You can check the value of JRE_HOME property in server.startup.sh and server.shutdown.sh files (Linux) or server.startup.bat and server.shutdown.bat files (Windows).
2. The maximum heap size for the Java virtual machine should be 6 GB or more: Change the heap setting in the server.startup.sh(.bat) file to JAVA_OPTS=$JAVA_OPTS -Xmx6000M
3. Use the –Xcompressedrefs argument for the Java virtual machine to reduce the 64-bit object size and more effectively use the available heap space. The tradeoff of this option is to limit the heap size to a maximum of 25 GB. However, most Java applications will not be affected by the limitation. Add the following property in the server.startup.sh(.bat) file: JAVA_OPTS=$JAVA_OPTS -Xcompressedrefs

Similar configurations are also optimal for WebSphere Application Server (Table 9). Note that we have not yet confirmed 50 concurrent users were supported by Rational Software Architect Design Manager Server deployed on Tomcat.

1. Set the thread pool property to 2 or 3 times the number of maximum concurrent users. For example, to support 50 concurrent users, in the WebSphere Application Server administrative console, under Application servers > your server name > Thread pools > WebContainer, set the value of the thread pool web container to 100-150.
2. Increase the maximum heap size of the Java virtual machine to 6 GB or higher. For example, in the WebSphere Application Server administrative console, change the maximum heap size in Application servers > your server name > Process definition > Java Virtual Machine to 6000 MB.
3. Use the –Xcompressedrefs option to better utilize memory and improve performance. For example, in Application servers > your server name > Process definition > Java Virtual Machine setting page, add the –Xcompressedrefs option to the Generic JVM arguments field.
4. If the operating system that hosts WebSphere Application Server is Linux, the maximum number of open files limit should be set to 50,000 or more. Otherwise the server cannot handle multiple user requests at the same time.

Table 9. WebSphere Application Server tuning

Storage requirements

Rational Software Architect Design Manager stores all contents and metadata in the database, and the indexes in the file system of Rational Software Architect Design Manager Server. The size of the database depends on the size of imported model projects and other repository activities, such as commenting, taking snapshots, and creating reviews. For example, the database size increased 1,472 MB after importing a 215 MB model project. The size of the database increases in a linear fashion that is based on the repository usage.

The size of the index also depends on the size of artifacts stored in the database. Large amounts of text and larger model files result in a larger index size. The index can grow to many times the size of database. Therefore, adequate disk space for the index is required. The index should be stored on a local drive to improve performance.

The disk space required by the database and index depends upon the size of model files, and the level of user activities. The minimal disk space requirement is 80 GB and for optimal performance the swap space should be twice the physical memory.

Network connectivity

Network connectivity in the Rational Software Architect Design Manager server environment should minimize latency between the application server and the database server (no more than 1 – 2 ms) and servers should be located on the same subnet. When using external storage solutions, the ideal is to minimize connectivity latencies and to achieve an optimal configuration by using optical fiber.

The latency between the server or workstation where the import engine is running and the application server also affects the performance of importing models. If the import engine is running on a server or workstation, which is not located on the same subnet as the application server, the average import speed is 0.38 MB per minute. The average import speed is 0.96 MB per minute if they are on the same subnet. For optimal performance, users should locate the computer running import engine and servers on the same subnet if the size of model projects being imported is very large.

Artifact sizing guidelines

The following values for artifact sizing ensure optimal performance of the repository when data sizes increase significantly.

Repository: There are no limits to the number of project areas and users if you use DB2, Oracle, or other supported database applications. Our test run was based on 50 registered users. Only 10 registered users are allowed if using an Apache Derby database.

Comments: You can have up to 5000 comments in the repository, which can also be deleted by the user with the project lead process role.

Concurrent users: Based on our performance test results mentioned in this document, Rational Software Architect Design Manager Server supports up to 50 concurrent users for configurations with 8 GB RAM and a 2-core CPU.

Search results: Up to 200 results will be returned when performing search on both the web client and Rational Software Architect client. A message prompts users to refine search keywords if more than 200 search results are returned.

Model import: There is no upper limit to the size of the model project being imported. The largest model project in our import performance testing was 156 MB, which contained 76,900 resources and 695,000 model elements, and which took 6 hours to import. The server’s performance can be affected when a model import is in progress. Therefore, for optimal performance, users should take advantage of the automated import process and schedule the import process when no users are active, such as after work or over the weekend. Multiple imports on one server simultaneously can also affect the server performance. For optimal performance, you should import models one by one.

For more information

• The Design Management project page on jazz.net
• Design Management Forum

About the authors

Claire Liu is a Software Verification Developer in the System Verification Test Team of IBM Rational Software Group. She was involved in developing end-to-end scenarios and testing multiple features for Rational Software Architect and its integration with other IBM products, including IBM Rational Team Concert, IBM Rational Requirement Composer, IBM InfoSphere Data Architect, IBM Rational ClearCase, and IBM Rational ClearQuest. Currently, Ms. Liu focuses on testing IBM Rational Software Architect Design Manager and the integration with Collaborative Lifecycle Management. Claire can be contacted at yliu@ca.ibm.com.

Ya Ju Yan is a Software Engineer in the System Verification Test Team of IBM Rational Software Group. She was responsible for performance testing for Rational Software Architect Design Manager. Recently she joined the Rational Insight System Verification Test Team and performs system and integration tests, which involve other IBM Rational products such as IBM Rational ClearQuest, IBM Rational ClearCase, IBM Rational RequisitePro, and other Jazz foundation products. Ya Ju can be contacted at yajuyan@cn.ibm.com.

© Copyright 2011 IBM