Collaborative Lifecycle Management performance report: RTC 4.0.6 release

Introduction

This report compares the performance of an unclustered Rational Team Concert version 4.0.6 deployment to the previous 4.0.5 release. The test objective is achieved in three steps:

• Run 4.0.6 RC1 (build CALM-I20140110-0300) with standard 4-hour test
• Run 4.0.5 GA with standard 4-hour test
• Compare the results and file any defects observed.

Disclaimer

The information in this document is distributed AS IS. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer’s ability to evaluate and integrate them into the customer’s operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk. Any pointers in this publication to external Web sites are provided for convenience only and do not in any manner serve as an endorsement of these Web sites. Any performance data contained in this document was determined in a controlled environment, and therefore, the results that may be obtained in other operating environments may vary significantly. Users of this document should verify the applicable data for their specific environment.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multi-programming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

This testing was done as a way to compare and characterize the differences in performance between different versions of the product. The results shown here should thus be looked at as a comparison of the contrasting performance between different versions, and not as an absolute benchmark of performance.

What our tests measure

We use predominantly automated tooling such as Rational Performance Tester (RPT) to simulate a workload normally generated by client software such as the Eclipse client or web browsers. All response times listed are those measured by our automated tooling and not a client.

The diagram below describes at a very high level which aspects of the entire end-to-end experience (human end-user to server and back again) that our performance tests simulate. The tests described in this article simulate a segment of the end-to-end transaction as indicated in the middle of the diagram. Performance tests are server-side and capture response times for this segment of the transaction.

Findings

Performance goals

• Verify that there are no performance regressions between current release and prior release with 1100/1200 concurrent users using the workload described below.

Findings

• RPT throughput is comparable in both 4.0.5 GA and 4.0.6 RC1.
• GC logs show similar trend in pause times except a couple in 4.0.6, shows no significant impact on the transaction rate. The gc collection trend in 4.0.5 GA and 4.0.6 RC1 are similar.
• Nmon data shows comparable system statistics in both 4.0.5 GA and 4.0.6 RC1 except in 4.0.5 GA Run 1 which is not reproducible.

Topology

The topology under test is based on Standard Topology (E1).

The specifications of machines under test are listed in the table below. Server tuning details listed in Appendix A

N/A: Not applicable.

Network connectivity

All server machines and test clients are located on the same subnet. The LAN has 1000 Mbps of maximum bandwidth and less than 0.2 ms latency in ping.

Data volume and shape

The artifacts were distributed between 2 components in 1 project, 50 scm streams and artifacts.

The repository contained the following data:

• 134 folders
• 983 files
• 50 streams

• Database size = 15 GB
• JTS index size = 1 MB
• CCM index size = 1 MB

Methodology

Test cases and workload characterization

This section describes in more detail what the RTC sequences do to emulate user activity.

RPPSequences are Java-based schedules that provide the setup and runtime iteration logic to drive a series of custom-code based "tasks" simulating user operations and use cases. Different user types are represented by named sequences to simulate activity by different user roles. A single user thread may be running at a higher than normal speed to simulate more than one user; for this reason each sequence has "factors" estimating how many individual tasks each user type would perform per hour and the estimated number of users is based on dividing total tasks for a user class by that "factor" to estimate number of users.

Communication between Developer and Build users rely on an intermediate API layer that calls an external process known as the "SCM Daemon" which handles the actual server connections; an agent usually has two of these daemons to distribute load. Contributor users include some local API calls as well as REST calls to simulate web browser traffic.

File Downloads:

Projects and Streams: Users are assigned to different streams in the same project to spread them out into small to medium teams working in each stream. The project/stream structure is up to the tester but is typically 10-50 streams.

Components and Workspaces: User workspaces work with one of two components for different operations:

• "Stable" component is fixed size and is not subject to changes or growth during the run. The file number and size are up to the tester but are typically 5 K to 10 K files with an average size of 10 Kbytes per file. This is used to get predictable behavior for Load operations.

• "Active" component usually starts empty but this is where Developer users create and modify files, so it grows quickly with a lot of users in the same stream.

Builds (BuildJavaSequence)

Currently one complete build cycle per hour represents one "build".

ONETIME SETUP:

• Create BuildDefinition, BuildEngine, and workspace as needed

RUNTIME LOOP:

BuildRequests: Simulate build engines by making multiple requests before build assignment by invoking the most commonly called webservices through the BuildToolkit at 6 different points in the sequence, doing the following each build cycle:

• 6 times 10x (getBuildDefinition)
• 6 times 30x (getBuildEngine, getNextRequest)
• 6 times 4x (setLastContactTime)

Build Cycle simulation: Send status messages and artifacts to the server simulating the communication process during real builds. Workspace Loading: Completely unload and load the workspace STABLE component once

Note: Longest operation

Start build: getBuildDefinition, getBuildEngine, requestAndStartBuild, getBuildResult

Activities: Each activity begings with startBuildActivity and ends with completeBuildActivity

PublishLinks: Publish (one URL)

Compile: Publish (compile.logs.xml, compile.logs)

PublishArtifacts: Publish (test-artifact.txt, 2x (smallzipfile.zip), link)

Testing: Publish (5 different results.xml files plus buildlog.txt)

Note: Time intensive because artifacts are parsed on client side and cause many web service calls

Complete: makeBuildComplete (signal build finished)

WorkItems (ContributorSequence)

The WorkItem Contributor sequence simulates a mix of activities in specific proportions. A single contributor is set to perform 20 tasks per hour in the RPP simulation. Workitem traffic consists of some REST calls isolated from early SVT HTTP script based tests and a series of API based tests that create and modify workitems, refresh workitem contents, and create, run, and delete queries.

ONETIME SETUP:

• Create initial workitem

RUNTIME LOOP:

Developer (DeveloperSequence)

Developers generally use the Eclipse rich client to download files, checkin changes, accept changes from other users and make deliveries; in addition they rely on Feeds data and perform Contributor operations. This sequence runs at 25 tasks per hour per developer. Each developer should be matched by a Feeds "user" at 40 tasks/hour and some additional Contributor traffic.

ONETIME SETUP:

• Create a new workspace and load the "Active" component used for making and receiving changes
• Create a private work directory of roughly 50 files and folders to use as a personal "scratch box" for making changes without conflicting with other users.

Note: Doing fresh setup periodically will probably smooth out longer runs

RUNTIME LOOP

Workspace operations

"During Day" activity loop (Runs 10 times within each runtime loop)

Feeds (FeedsSequence)

Feeds simulate a feature of the Eclipse rich client which checks the server for recent workitem or build events of interest to the user. Feeds are normally repeated for each user every 10 minutes for the projects they are working with. RTC Development early on indicated that they had seen roughly 40 feeds operations per developer (Eclipse user) per hour and that is the basis for our simulation rate.

ONETIME SETUP:

Create one feed channel for the current project and one feed channel for the associated team area. Usually sets up two feeds channels per user; RTC developers average around eight.

RUNTIME LOOP:

RefreshFeeds: Called repeatedly during run

Request each feed channel parse its input (make a query to the server and parse results)

Reset time window: Simulate how client works by creating a new dummy news item to reset the time window subsequent feeds requests should use. Without doing this, all feeds produce news from the beginning of time.

Results

User Estimation

Task throughput

Tasks represents use-cases based on frequently done standard user operations. Task throughput is the number of completed tasks throughout the run duration.

CCM Server web services

Server side web services data are collected from the CCM server using JazzMon / JTSMon monitoring tool.

System comparison

OS resource usage - average CPU

OS resource usage - average memory

Garbage collection

Verbose garbage collection is enable to create the GC logs. The GC logs shows very little variation between runs except at 30 min duration there is a long pause time in 4.0.5 GA. There is also no discernible difference between versions.

Below is one example of the output from the GC log.

4.0.5 GA CCM garbage collection

4.0.6 RC1 CCM garbage collection

4.0.5 GA JTS garbage collection

4.0.6 RC1 JTS garbage collection

Appendix A

-Xmn768m -Xmx6g -Xms6g -Xgcpolicy:gencon -Xgc:preferredHeapBase=0x100000000 -Xloaminimum0.02 -Xsoftrefthreshold5 -Xdump:stack:events=fullgc 
-Xdump:java:events=systhrow+user,filter=java/lang /OutOfMemoryError,request=exclusive+prepwalk 
-Xdump:stack:events=allocation,filter=#20m,request= -XX:MaxDirectMemorySize=1500M

Appendix B - References

• Comparison of 4.0.5 vs 4.0.4
• Comparison of 4.0.4 vs 4.0.3
• Comparison of 4.0.2 RC1 vs 4.0.3 M4 CALM-I20130411-0627

Appendix C - History

Prior reports are located on jazz.net development wiki:

• 4.x: https://jazz.net/wiki/bin/view/Main/CLMBaselines2012
• 3.x: https://jazz.net/wiki/bin/view/Main/CLM301Performance

About the authors

ChetnaWarade is has been a Software Engineer at IBM for over 9 years. She is part of the Rational Performance Engineering group.

Questions and comments:

• What other performance information would you like to see here?
• Do you have performance scenarios to share?
• Do you have scenarios that are not addressed in documentation?
• Where are you having problems in performance?

Warning: Can't find topic Deployment.PerformanceDatasheetReaderComments