Non-functional deployment considerations when adopting IBM Engineering Lifecycle Management solution for global configuration management.
Tim Feeney, IBM; Paul Ellis, IBM; Breun Reed, Persistent Systems
Last updated: 1 May 2020
Build basis: IBM ELM Solution 7.0
Introduction
For years, source code management has embraced configuration management and the ability to use versioned artifacts in multiple streams and baselines. Since version 6.0 of the IBM Engineering Lifecycle Management (ELM) solution, the requirement, design, and quality management applications offer configuration management. This article explores the additional non-functional deployment considerations when adopting an ELM solution for global configuration management.
There are many aspects to consider for each of the new applications that perform the underlying functions behind configuration management. There are varying levels of value to be gained based on your goals and where you are struggling. The IBM ELM solution provides value from baselines, to change management, parallel development, product variants, and product line engineering.
After you determine whether your organization would benefit from configuration management, you need to evaluate your existing processes and determine how they will change to reflect the new concepts and operations. Additional information is available within CLM configuration management: Adoption guidance and practices on the conceptual aspects of this topic. See the full set of related articles in this RSS feed.
This article guides you through important deployment considerations for adopting configuration management across your ELM applications. If you are new to the concepts, capabilities, and terminology of configuration management, read this help topic in IBM Knowledge Center. If you need assistance with adopting configuration management, ask your IBM representative about service offerings that can help you.
In this article:
- We begin by presenting and discussing the deployment topology implications and recommendations.
- We then describe the important performance and scalability considerations.
- Finally, this topic is not complete without a discussion of serviceability capabilities which are all the more vital in complex configuration management enabled deployments.
Topology implications and recommendations
Configuration Management for the ELM solution requires the deployment of three additional applications: the link Index (commonly referred to as LDX), Global Configuration Management (GCM), and Lifecycle Query Engine (LQE). Learn more about these essential applications and configuration management concepts here: Overview of Configuration Management.
When deploying ELM in production for use with configuration management, an enterprise or federated topology should be used. A department topology is insufficient and should not be used in production, or with production levels of data, or for performance testing with configuration management.
By default, LDX is co-located with the Jazz Team Server (JTS). When co-locating LDX or GCM on existing servers, additional CPU/Memory resources may be needed; when doing so, each should be in their own WAS/Liberty profile. However, for large, complex configuration management enabled deployments we advise that the LDX and GCM applications each be on their server.
For multiple instances of ELM, such as a federated topology with multiple JTS instances, only a single instance of LDX must be used. At this time, we do not support deploying multiple LDX instances for any environment that contains multiple GCM instances, since we require the use of a single LDX across all applications that contribute directly or indirectly to any given global configuration.
Both the LDX and LQE applications utilize Apache Jena index technology, which works best if there is sufficient memory to hold the index in memory and the underlying storage for the index is as fast as possible, such as an SSD.
Multi-server IBM Engineering Requirements Management DOORS Next and LDX
In a federated or enterprise deployment topology, there are cases where you may need to enable cross-project and cross-component linking between IBM Engineering Requirements Management DOORS Next (DOORS Next) servers. This is increasingly important to customers who need to link versioned artifacts between many product lines. To link these artifacts, the DOORS Next resources for each server must be indexed into a single common LDX instance. When deploying this advanced configuration, the LDX should be on its server given the expected higher volume of interactions between DOORS Next and LDX/JTS. Although LDX works fine on a Windows-based system, due to limitations in I/O throughput, we have seen customers achieve better success moving to Linux or running LDX on physical rather than virtual servers.
Additionally, you may need to split your DOORS Next artifacts across two (or more) servers as you grow in scale. Here is a topology example of multiple application server patterns. The latest sizing and tuning recommendations can be found here: 7.0 performance: IBM Engineering Requirements Management DOORS Next
Collaborating GCMs
This is an advanced configuration included here for completeness. Clients interested in this capability should consult with IBM to fully understand its benefits, limitations, and administrative trade-offs before its adoption.
In a federated topology that contains two or more product lines, there may be multiple instances of GCM. In ELM 6.0.6, we introduced the concept of local and remote Global Configurations that can enable what is called an “external contribution.” Clients would be interested in this if they have an immediate need to collaborate across business units/product lines.
Linking and collaboration across remote GCM instances also require IBM MessageSight, a Message Queuing Telemetry Transport (MQTT) Broker (see MessageSight Deployment Options) the section MQTT deployment considerations section below for additional details).
For more information on the constraints, limitations, enabling and other related topics, please review Enabling GCM servers to contribute configurations to other GCM servers.
Configuration-aware reporting
As the size and complexity of data and users managed by the IBM ELM solution grow, along with the reports and queries needed to provide critical data insights, single servers for LQE and RB might be insufficient. There are patterns and techniques for achieving greater data, user, and query scale. These are detailed in Scaling the Configuration-aware Reporting Environment.
Performance and scalability
There are many variables to consider when understanding the performance and scalability of the IBM ELM solution. The best source of documented advice is in the jazz.net Deployment wiki. The principle considerations when planning for performance are to assess your fundamentals such as:
- Number of concurrent users expected at peak times
- Workloads being performed in the background
- Data set shape, sizes, and growth over time
- Server sizing (number of cores, size of RAM, disk storage speed)
The use of global configuration management will alter these characteristics. The most notable considerations specific to global configuration management are the three new applications, described earlier, each with specific needs in terms of resources required to run optimally and also scalability.
The impact on the main ELM applications, IBM Engineering Workflow Management (EWM), DOORS Next, and IBM Engineering Test Management (ETM) are quite different, mainly due to the introduction of versioning of DOORS Next and ETM requirements and test artifacts.
For optimal performance using global configuration management, IBM recommends clients deploy 7.0 (latest iFix). There have been significant improvements made in the 7.x release stream to improve scale and reliability.
For optimal performance when using global configuration management, we recommend the use of Oracle or IBM Db2 over Microsoft SQL Server. See What’s new in IBM Engineering Lifecycle Management 7.0 enterprise deployments.
Note that performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multi-programming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
ETM
The latest performance test report for ETM with configuration management enabled details scalability to 10M artifacts and 1000 users.
DOORS Next
Recent tests of DOORS Next included 20 million artifacts per repository on Oracle (10 million on Db2) and 3.8M per project area (2M on Db2). Test results are described in 7.0 performance: IBM Engineering Requirements Management DOORS Next.
Load for DOORS Next has shifted to the database server in 7.0, which can impact deployments where multiple RM servers share a single database server. We recommend each DOORS Next server not share database resources with other applications.
DOORS Next performance is sensitive to the number of contributions in a global configuration, especially on Db2. On Db2, some use cases may be slow in global configurations with 1250 or more contributions. Performance is not sensitive to the number of components per project or the total number of components.
Note for those remaining on DOORS Next 6.x, be advised of its lower sizing limits and recommended configuration cache and other tuning called out in Rational DOORS Next Generation: Organizing requirements for best performance.
Rhapsody Model Manager
As of 7.0, a Rhapsody Model Manager (RMM) server can be combined with an EWM server, more specifically, the Architecture Management (AM) capability can be deployed as an extension to the CCM application. Recent tests compared average EWM service times when running a workflow consisting of EWM only scenarios versus combined EWM+AM workflows. Test results in Performance impact of combining Rhapsody Model Manager and Engineering Workflow Management servers show little impact to EWM service times. Future articles will guide the trade-offs related to deploying separate or combined servers. At this time, there are no known data shape sensitivities concerning RMM and global configuration management performance. All EWM SCM limits apply such as the recommendation to limit components in a stream to 2500 and files/folders to 100,000 in a component.
Serviceability considerations
With the additional required applications and higher resource demands when using global configuration management, it imperative that you have a clear, thought-out serviceability strategy for IBM ELM deployment environments. At a minimum, this includes enterprise monitoring, awareness of resource-intensive scenarios, and an understanding of how to troubleshoot common anomalies.
Enterprise monitoring
Having an enterprise monitoring strategy is a best practice for IBM ELM deployments and recommended for any deployment involving our GCM capability.
The IBM ELM solution supports enterprise monitoring as part of our serviceability strategy. Applications provide detailed instrumentation of data useful for proactive and reactive monitoring and management of an IBM ELM deployment. The instrumented data is provided in the form of Java Management Extensions (JMX) MBeans, published by the applications and available for collection and analysis by an enterprise monitoring application.
Useful documentation and guidance for our MBeans:
If you are getting started with monitoring, focus on the set of MBeans described in CLM Monitoring Primer. There are many more defined in the MBeans Reference List. Once you’ve implemented the base set, you can expand from there. Proactively monitor by setting recommended thresholds with corresponding alert notifications to prompt further investigation. Thresholds may need to be adjusted over time based on experience. Recommend monitoring normal operations to establish appropriate baselines and adjust thresholds accordingly to reduce false-negative alerts. Some monitoring tools can use machine learning and statistical analysis to adapt thresholds.
Scaling the Configuration-aware Reporting Environment has some useful tips on monitoring the LQE server to determine if it is operating well or if sizing or topology changes are warranted.
When monitoring your ELM environment, be sure to take an enterprise-wide perspective as application scenarios often span applications and a symptom may be experienced in one application but caused by another. In this case, some correlation of data from multiple application resources may be needed.
Resource-intensive scenarios
There are some scenarios when using the IBM ELM solution that is known to drive load on an application server that could lead to outages or overall diminish the quality of service for end-users. These are called resource-intensive scenarios. It is good practice to know what these are so that your teams’ usage of an application can be adjusted accordingly. Concerning configuration management, there are some resource-intensive scenarios related to the use of global configuration management and lifecycle query engine applications. Each scenario occurrence is logged in the respective application log and is recorded in Expensive Scenario Details, Resource-intensive scenario elapsed, and summary time detail application MBeans.
For more information
- Getting to a right-sized Jazz environment
- Best practices for using global configuration management
- Performance datasheets and sizing guidelines
- Standard topologies overview
- Resource-intensive scenarios
About the authors
Tim Feeney is a Senior Solution Architect who works with IBM’s key customers to architect lifecycle solutions, deployment topologies and usage models across the IBM Engineering Lifecycle Management solution; he uses these experiences to develop best practices and guidance and works with IBM Offering Management and Development to further improve the solution. Tim can be contacted at tfeeney@us.ibm.com.
Paul Ellis is an Executive Technical Specialist who works as part of a global response team within IBM Engineering Client Success. Paul is predominantly focused on complex deployment, performance, and other enterprise-client issues, how to improve the Client Success experience, and overall product serviceability. Paul can be contacted at paul.ellis@ie.ibm.com.
Breun Reed is a Senior System Test Architect for Persistent Systems, Inc. covering IBM’s ELM solution. His primary test focus is the end to end enterprise customer use case validation covering by not limited to scalability, upgrade, migration, administration, security, licensing, monitoring and deployment of complex ELM topologies. Breun can be reached at breun.reed@persistent.com.
© Copyright IBM Corporation 2020