r17 - 2015-10-01 - 13:42:20 - StevenBeardYou are here: TWiki >  Deployment Web > DeploymentPlanningAndDesign > CLMSizingStrategy

CLM Sizing Strategy

Authors: StevenBeard, GrantCovell, PoornimaSeetharamaiah
Build basis: CLM and SSE 4.0.6 and later
Date: April 11, 2014

Introduction

Whether new users or seasoned experts, customers using IBM Jazz products all want the same thing: They want to use the Jazz products without worrying that their deployment implementation will slow them down, that it will keep up with them as they add users and grow. A frequent question we hear, whether itís from a new administrator setting up Collaborative Lifecycle Management (CLM) for the first time, or an experienced administrator tuning their Systems and Software Engineering (SSE) toolset, is ďHow many users will my environment support?Ē

Back when Rational Team Concert (RTC) was in its infancy we built a comprehensive performance test environment based on what we thought was a representative workload. It was in fact based upon the workload the RTC and Jazz teams itself used to develop the product. We published what we learned in our first Sizing Guide. Later sizing guides include: Collaborative Lifecycle Management 2011 Sizing Guide and Collaborative Lifecycle Management 2012 Sizing Report (Standard Topology E1). As features were added and the release grew, we started to hear about what folks were doing in the field. The Jazz products, RTC especially, are so flexible that customers were using them with wonderfully different workloads than we had anticipated.

Consequently, we stepped back from proclaiming a one-size fits all approach, and moved to presenting case studies and specific test reports about the user workload simulations and the loads we tested. We have published these reports on the jazz.net Deployment wiki at Performance datasheets. We have tried to make a distinction between performance reports and sizing guides. Performance reports document a specific test with defined hardware, datashape and workload, whereas sizing guides suggest patterns or categories of hardware, datashape and workload. Sizing reports are not specific and general descriptions of topologies and estimations of workloads they may support.

Throughout the many 4.0.x release cycles, we were still asked ďHow many users will my environment support?Ē Our reluctance to answer this apparently straightforward question frustrated customers new and old. Everyone thinks that as the Jazz experts we should know how to size our products. Finally, after some analysis and messing up countless whiteboards, we would like to present some sizing strategies and advice for the front-end applications in the Jazz platform: Rational Team Concert (RTC), Rational Requirements Composer (RRC)/Rational DOORS Next Generation (DNG) and Rational Quality Manager (RQM). These recommendations are based upon our product testing and analysis of customer deployments.

Standard disclaimer

A sizing estimate is an approximation of the hardware or software resources required to support an application set that is either currently implemented or new. The level of effort and scope of sizing degree of variability can range from small to very significant. Sizing does not constitute a performance guarantee. Customer results may vary, and IBM assumes no liability for actual results that differ from any estimates provided by IBM.

Where does this data come from?

The data in this sizing advice document comes from tests executed by the Rational Performance Engineering team and hardware and sizing models generated by IBM Global Techline, an internal pre-sales technical support organization. Further refinements of user patterns are derived from our analysis of many customer deployments. Specific references to the relevant data and test reports are cited below.

Basic principles

  • The jazz products are wonderfully flexible. Whether you are new to Agile or a continuous integration junkie, the Jazz products will make planning, tracking, collaborating and delivering your project easier than ever before. They can be deployed in variety of configurations and they can be used in any number of ways. They can be used by large teams and small teams, co-located teams or widely distributed teams.
  • What works for one customer might not work for another. Each organization has its own processes, rules, templates and habits. Each organization has valuable employees with varied skills and experiences. The workflow and datashape in use at one company will probably not meet the needs and demands of any other company. Each company has rules about how systems and hardware are configured and maintained, and one organizationís compliance may not be appropriate for anotherís. Every organization has unique network characteristics, user authentication, VPNs and customized desktop configurations. Given all these and possibly other innate differences, the identical hardware deployed at one customer cannot be guaranteed to behave similarly to the exact deployment at another customer.
  • It is normal for a customerís workload to change during a project lifecycle. As end-users discover their optimal ways to work with the flexible Jazz tools, itís not uncommon that the way they interact with the Jazz products changes over time. It isnít just that more requirements are created at the beginning of a project, or that more defects are sometimes logged towards the end, itís that as teams practice Agile and get used to iterative development, they work more collaboratively and leverage the strengths of the Jazz platform. For example, in some organizations, the number of work items in plans decrease, however work items increase their parent-child relationships.

Recommended topologies

We have outlined recommended deployment topologies for the CLM and SSE products on the jazz.net Deployment wiki. Customers implementing the Rational solution for Collaborative Lifecycle Management (CLM) or the Rational solution for Systems and Software Engineering (SSE) often ask how they should deploy the tools so that they perform well, run robustly, and can evolve without restriction. To simplify the wide range of choices, the Standard Topologies article series outline several standard topologies which are the expected and most frequently chosen deployment patterns.

Minimum requirements and practical limitations

We suggest an ideal minimum of 4 logical processors and 8 GB RAM on a 64-bit machine. The absolute minimum is 4 GB of RAM on a 64-bit machine, but for an enterprise deployment we prefer to suggest the ideal minimum. Also see System Requirements for Collaborative Lifecycle Management 4.0.5 and 4.0.6 and System Requirements for Systems and Software Engineering 4.0.5 and 4.0.6.

You will notice that the published test reports utilize hardware (cores and RAM) which are sized generously. A well designed deployment will be able to handle peak loads and unexpected usage patterns. We customarily size environments so that they may be able to accommodate bursts of up to double the amount of expected normal load. However, we do not recommending presuming an environment can handle such extreme loads for more than 4% of their standard operational time.

A deploymentís performance can be influenced by many things

The performance and behavior of a CLM or SSE deployment can be influenced by many factors or dimensions. These dimensions fall into two categories: application and product related (functional dimensions) and environment or topology related (non-functional dimensions). CLM and SSEís flexibility and performance can be undermined by unintentional functional and non-functional design choices.

It is easy to design an ideal topology which presumes zero-second latency, unlimited bandwidth and ever-increasing storage capacity, but such capabilities are impossible in real deployments where there are known and possibly unexpected constraints and limits. It is important to be mindful of the gamut of possible constraints for the various dimensions of a CLM and SSE environment.

This table collects some of the factors which may influence performance. This table is by no means comprehensive. There may be other factors not listed here. A factor listed here may have different effect in different environments.

Dimension Factors unique to every customer deployment which may influence performance
Product specific
  • Customized templates and workflows
  • Number of work items in a plan (e.g.: more than 3000)
  • Number and type of widgets on a dashboard (e.g.: more than a dozen plan widgets; widgets which load charts)
  • Number of tabs on a dashboard (e.g.: dashboards with many widgets that show graphs and BIRT reports)
  • Complex plan views (e.g.: plans which also show Gantt charts)
  • Frequent feed requests
  • Complex queries requested against many requirements
  • Clients, servers and build agents at different version levels (e.g.: clients at version 3.x and servers at 4.x)
Network
  • Latency is greater than 200 ms more than 10% of the time
  • Bandwidth less than 1 Gbit
  • Packet loss greater than 2%
  • Datacenter not co-located with application servers
Storage
  • Local storage
  • Slow I/O to disc
  • SAN or NAS not connected by fibre channel (or any faster protocol)
  • Datacenter not co-located with application servers
Browser clients
  • Internet Explorer version 8 or earlier
  • Browser not updated
  • Browser not running most current version of java
  • Browser not HTML 5 compatible
Desktop clients
  • Eclipse shell integrated with multiple dependencies
Build agents
  • Uncontrolled and unmonitored use of build agents
Application server
  • If virtualized, resources (CPU, RAM, network) are not properly reserved or dedicated; entitlement is not guaranteed; shared pool does not have enough resources
  • Undersized CPU
  • Undersized RAM
  • OS not up to date
  • Not co-located with database
Database server
  • If virtualized, resources (CPU, RAM, network) are not properly reserved or dedicated; entitlement is not guaranteed; shared pool does not have enough resources
  • Undersized CPU
  • Undersized RAM
  • OS not up to date
  • Not co-located with application server
  • Not properly indexed
  • Backups occur during peak usage times
License server
  • Not co-located with JTS server
  • Latency between license server and JTS server is more than 50ms
Authentication server
  • Does not meet an industry standard
Load balancer
  • No front-ending load balancer (neither software nor hardware)
  • Not using a reverse proxy

Business continuity
  • No high availability or disaster recovery policy
  • High availability or disaster recovery policy not practiced
Users
  • More users than anticipated
  • Users work differently than anticipated
  • User population has different roles than anticipated
  • Users work with datashapes (documents, files, code) which are larger in size than anticipated
  • User population is not geographically distributed (e.g.: all users commonly use the environment at set schedules, or are expected to login near coincidentally at the start of a workday)
Organizational Maturity
  • More mature teams (greater degree of Agile experience or even a highly developed waterfall process) tend to use a system more intensively

Given assumptions and caveats

  • All hardware and operating systems are 64-bit.
  • Representative product workloads (examples of how teams actually use the tools and what sort of load and transactions may occur) are described within the Rational Performance Engineering datasheets accessible here.
  • For these examples, each application (Rational Team Concert, Rational Requirements Composer/Rational DOORS Next Generation, and Rational Quality Manager) are hosted on a dedicated server and connected to a separate Jazz Team Server (JTS). This document does not address how the Jazz Team Server (JTS) scales. JTS scaling information can be found here.
  • Sizing estimates are provided for an approximate range of concurrent or active users. Decreasing the range of estimated concurrent users does not equate to a corresponding decrease in estimated resources. In other words, if an estimate provides a range of 300 to 500 concurrent users, it cannot be assumed that reducing by half the number of cores and RAM will meet the needs of half the amount of estimated users. In fact, we would recommend the same hardware configuration whether estimating for 150 to 250 users or 300 to 500 concurrent users.
  • Similarly, increasing the range of estimated concurrent users does not equate to a corresponding increase in estimated resources. In other words, if an estimate provides a range of 300 to 500 concurrent users, it cannot be assumed that doubling the amount of cores and RAM will meet the needs of twice as much concurrent users.
  • We cannot estimate the size of database repositories. The size of the database is governed by many factors: The organizationís customized workload, the organizationís customized datashape, and the work habits of the people in the organization. We suggest estimating a generous starting size for your database repositories, but more importantly, we suggest monitoring the growth of the repositories. Your database will grow based upon how it is used in your organization, and understanding your trends and growth patterns is the best way to anticipate database sizing needs. It is best to consider an enterprise storage solution which supports a high IOPS throughput since IO directly impacts performance. In addition mature storage solutions have the ability to size volumes as needed and optimize storage needs with real-time data deduplication.
  • RAM consumption should never be more than half of the total available RAM; Maximum JVM heap size should not exceed half of the serverís allocated RAM.
  • You may note that the data presented here may be different from data previously presented or even data that may be presented in the future. This sizing advice does not replace or supersede any other advice. Sizing estimates are derived from actual test data and based upon assumptions and for the most accurate sizing advice for your situation you need to work with IBM to understand your unique requirements.
  • This document considers single applications connected to a single JTS server. Multiple application servers can be connected to a single JTS, and sizing advice for such configurations will be considered in a future publication. Considerations and strategies for when multiple application servers should be used will be discussed in a future publication.

Registered and concurrent user definitions

When we talk about users, we generally talk about registered or licensed users and a subset of concurrent or active users. Registered users refer to the number of users who are licensed and permitted to use a system. A company of 1200 employees may have 800 registered users. Concurrent users are the number of users logged in and actually using the system at any given time. A company of 1200 employees may have 800 registered users, but only 200 are logged in and actively working at any given time. We call these 200 users concurrent and active regardless of the intensity of how they may be working.

The number of concurrent users can often be estimated from the number of registered users. In an organization located on one continent or spanning just a few time zones, the number of concurrent users can be estimated to be between 20% and 25% of the total number of registered users. In an organization which spans the globe where registered users work in many different time zones, the number of concurrent users can be estimated to be between 10% and 15% of the total number of registered users.

For now we discuss server load in terms of users. This can be potentially misleading as different organizationsí users will work differently. We plan to add more precise transaction-rate and workload-rate data to our reports and sizing estimates, but for now, we specify server load in terms of users.

Rational Team Concert (RTC): Examples of different RTC usage patterns

Of all the Jazz products, Rational Team Concert (RTC) permits the widest variety of end-user and automated workflows. When estimating a RTC workload it is important to consider what percentage of the anticipated usage will come from work item users, SCM users and build agents or any combination of the three. Description of representative RTC workload we use in our sizing estimates can be found here.

Rational Team Concert (RTC) has three major workload types which can be used concurrently:

  • Work item
  • Software configuration management (SCM)
  • Build agent

Our representative workloads combine these major workload types:

  1. Work item only workload
  2. Combined work item and SCM workload
  3. Combined workitem, SCM and build agent workload

Rational Team Concert (RTC): A: Work item only workload

Given this representative base hardware
(configured as hypervisor):
Model x3550 M4
Processor Intel XEON processor E5-2670v2
Processor speed 2.5 Ghz
And given a virtual machine (VM) of this size: Logical cores 4 vCPUs
RAM 8 GB or more
We estimate that this workload will perform comfortably Approximate range of concurrent work item users: Between 400 and 600

Rational Team Concert (RTC): B: Combined work item and SCM workload

Given this representative base hardware
(configured as hypervisor):
Model x3550 M4
Processor Intel XEON processor E5-2670v2
Processor speed 2.5 Ghz
And given a virtual machine (VM) of this size: Logical cores 4 vCPUs
RAM 12 GB or more
We estimate that this workload will perform comfortably Approximate range of concurrent work item users: Between 200 and 300
Approximate range of concurrent SCM users: Between 100 and 200

Rational Team Concert (RTC): C: Combined work item, SCM and build agent workload

Given this representative base hardware
(configured as hypervisor):
Model x3550 M4
Processor Intel XEON processor E5-2670v2
Processor speed 2.5 Ghz
And given a virtual machine (VM) of this size: Logical cores 8 vCPUs
RAM 16 GB or more
We estimate that this workload will perform comfortably Approximate range of concurrent work item users: Between 200 and 300
Approximate range of concurrent SCM users: Between 100 and 200
Approximate range of concurrent build agents (presuming all clients, servers and build agents are at same version level): Between 100 and 200

Rational Requirements Composer (RRC) / Rational Doors Next Generation (RDNG)

Description of representative RRC workload we use in our sizing estimates can be found here.

Given this representative base hardware
(configured as hypervisor):
Model x3550 M4
Processor Intel XEON processor E5-2670v2
Processor speed 2.5 Ghz
And given a virtual machine (VM) of this size: Logical cores 4 vCPUs
RAM 8 GB or more
We estimate that this workload will perform comfortably Approximate range of concurrent RRC/RDNG users: Between 300 and 400

Rational Quality Manager (RQM)

Description of representative RQM workload we use in our sizing estimates can be found here.

Given this representative base hardware
(configured as hypervisor):
Model x3550 M4
Processor Intel XEON processor E5-2670v2
Processor speed 2.5 Ghz
And given a virtual machine (VM) of this size: Logical cores 4 vCPUs
RAM 8 GB or more
We estimate that this workload will perform comfortably Approximate range of concurrent RQM users: Between 350 and 500

Lifecycle Query Engine (LQE)

Given this representative base hardware
(configured as hypervisor):
Model x3690 X5
Processor Intel XEON processor E7-2830
Processor speed 2.13 Ghz
Ideal Minimum Generic Evaluation Topology: Logical cores 8 vCPUs
RAM 16 GB or more
Ideal Minimum Departmental Topology: Logical cores 24 vCPUs
RAM 96 GB or more
Ideal Minimum Enterprise Topology: Logical cores 32 vCPUs
RAM 256 GB or more

What is a sizing estimate for anyway? What are the next steps?

A sizing estimate attempts to forecast the number of users which a prescribed hardware configuration may comfortably support with a maximum response time at a defined interaction rate and a typical datashape. A sizing document is often created before a product is deployed in order to estimate the anticipated hardware needs for a deployment. Sometimes the sizing documentís most immediate goal is to specify hardware for allocation or procurement. Very often the sizing document specifies a maximum or largest anticipated target number of users simply so that the corresponding hardware allocation will not be exceeded and the anticipated hardware is requested only once.

Every CLM sizing document is an estimate: The CLM products are flexible and can be deployed to meet customersí needs; customer hardware and deployment environments are unique for each customer; customer workload and habits will be unique and can be expected to vary across the productsí deployment lifetime.

sizinglifecycle.png

A sizing estimate represents the best-guess assessment at a given point in time. As an environment is uses and grows, it is natural that assumptions made to support the sizing estimate may change. After deploying the environment we strongly recommend monitoring the environment and re-evaluating the assumptions and conclusions made during the planning phase as part of a deploymentís maintenance strategy. It is typical that an environment will grow and mature differently than the sizing estimate may anticipate. A standard maintenance and monitoring plan should include revisiting the sizing estimate and appropriately adjusting and tuning your deployment at key milestones, if not at a yearly minimum.

References

IBM System x3550 M4, http://www-03.ibm.com/systems/x/hardware/rack/x3550m4/

Standard deployment topologies overview , Steven Beard, Grant Covell, Tim Feeney, David Chadwick, Vaughn Rokosz: https://jazz.net/wiki/bin/view/Deployment/StandardTopologiesOverview

Performance impact of sharing a Jazz Team Server, Joseph Stellrecht, Vaughn Rokosz; July 24, 2013: https://jazz.net/wiki/bin/view/Deployment/PerformanceImpactOfSharingAJazzTeamServer

Collaborative Lifecycle Management 2012 Sizing Report (Standard Topology E1), Vaughn Rokosz, Skye Bischoff, David Chadwick, and Tim Feeney; June 13, 2012: https://jazz.net/library/article/814/

Collaborative Lifecycle Management 2011 Sizing Guide, David Chadwick, Tim Feeney, Philippe Mulet, Anuradha Ramamoorthy, and Skye Bischoff; June 24, 2011: https://jazz.net/library/article/641

Rational Team Concert (RTC) 2.0 sizing guide, Paul W. Weiss, Scott Rich, Jean-Michel Lemieux; June 24, 2009: https://jazz.net/library/content/articles/rtc/2.0/enterprise-configuration/sizing-guide/index.html

System Requirements for Collaborative Lifecycle Management 4.0.5 and 4.0.6, Deborah Weil-O'Day: https://jazz.net/wiki/bin/view/Deployment/CLMSystemRequirements405406

System Requirements for Systems and Software Engineering 4.0.5 and 4.0.6, Cindy McKeen: https://jazz.net/wiki/bin/view/Deployment/SSESystemRequirements405


Questions and comments:
  • What other sizing guidance would you like to see here?
  • Are there situations and topologies which are not covered?

  • I think we need to add JRS and DCC to the mixe -- PhilippeChevalier - 2016-01-11 - 10:31
  • "If there is too much demand for memory, the operating system's file cache may shrink and this can impact the caching of index data used by CLM applications. This is especially true for DOORS Next Generation and the Lifecycle Query Engine."

When we write "cache" and "caching" here, I think the Linux terminology may actually be "buffer" and "buffering".

In the Kista support team right now, we have some discussions about how you can monitor the memory usage for indexing. Focusing on active memory or process memory consumption, will not show how the OS buffers indices in RAM to avoid disk i/o. -- ErikMats - 2016-03-16 - 05:56

  • Please add JRS sizing guide -- DianeEveritt - 2016-09-28 - 09:10

Edit | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r17 < r16 < r15 < r14 < r13 | More topic actions
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.
Ideas, requests, problems regarding the Deployment wiki? Create a new task in the RTC Deployment wiki project