This article provides you with a list of actions you should be taking when planning an upgrade. It is intended to help you think through the upgrade process without getting lost in the execution details.
The following advice has been adapted from a presentation at
InterConnect 2015, 2016 and 2017 and is intended to be release agnostic. However, where links are release specific and pertinent then the links are for the latest 6.0.x release.
The 3 core aspects to an upgrade
Over the past number of years the IBM teams, that focus on Collaborative Lifecycle Management(CLM) upgrades, have realized that there are essentially 3 core elements to completing a successful upgrade. The technical aspects of an upgrade, whilst important, are certainly not the most critical.
This article details what lies behind the 3 core aspects of an upgrade, from planning through to execution.
Planning
"What considerations should be in my plan?":
People
There are essentially three sets of people who are involved during the upgrade lifecyle. The Users and the various stakeholders are the most obvious. However, as an upgrade is so much more than the technical steps on the day/weekend, utilizing project management techniques is strongly advised for larger deployments.
Also consider which IT teams need to be involved during the upgrade, who owns each of the systems, who owns each operation
- application server admin
- database admin
- networking admin
- don’t forget the training aspect
Documentation
The most important document is the
Interactive Upgrade Guide (IUG), where you can generate instructions tailored to your configuration. The IUG asks that you specify your configuration, operating system, application(s) to be upgraded, data warehouse requirements, integrations, etc and uses this input to generate detailed instructions and an upgrade checklist specifically tailored for your configuration.
Also ensure you utilize:
The support portal and jazz.net sizings are generally used as a basis for sizing based on the typical requirements stated. Plan to have the available capability to expand and explore past the minimum requirements if possible. When sizing, consider your future expansion needs and most stressful usage scenarios to determine the needed capacity and topology.
Timings
Ensure that there is plenty of contingency within an upgrade plan. Add a time buffer to each major operation to allow for unforeseen problems.
Depending on your data, some of the upgrade processes can take quite a bit of time and if a problem is discovered, the upgrade may need to be restarted.
- The best upgrade plans, will state the hand-off between operations and how these steps will be validated
- Plan for your applications to be unavailable
- Someone must manage the various parties and the overall progress
- Ultimately, if the upgrade does not go to plan, there must be a focal point that determines if the upgrade is cancelled or whether there is sufficient contingency not to roll-back
Features
New features help support your business case for upgrade. There are several pages on jazz.net that are designed to assist you with understanding the release you're looking to adopt.
- What is new in CLM
- The Overview tab in the product/solution downloads page provides a summary of the main features along with video demonstrations.
- For further details, look at the New and Noteworthy section on the product downloads page
- Release Notes states what was fixed in the release, enhancements, workarounds, current open issues, etc.
When reviewing features of the CLM application, ensure also that you look at the each of these pages for each of the applications you wish to upgrade
- Verify that your hot fixes are in the release
- Verify that all system requirements are met, required integrations are supported, and product release levels are compatible
Environments
Unless you are running as a managed service, or in the cloud, then Collaborative Lifecycle Management (CLM) is a software stack with many considerations. Where multiple components require changing, you need to understand in which order to execute these.
- Ensure any customizations/settings are brought forward, for example: Java Virtual Machine heap size changes
Approach
If there are multiple instances of CLM, or any particular application involved, then that invariably also means multiple instances of users, stakeholders and considerations to balance. This should dictate your approach, whether it be one colossal upgrade effort for all, or a more segmented piecemeal option. The interested parties may make this simpler for you by not being able to upgrade simultaneously due to conflicting release schedules etc. Embrace the approach most suited to you and your stakeholders.
Example scenario
In terms of selecting which component to upgrade first, see how an upgrade from CLM 4.0.3 to 5.0.2 could be approached
The operating systems below may be used to host CLM applications:
|
Before |
After |
CLM Version |
4.0.3 |
5.0.2 |
Database version |
DB2 9.7 |
DB2 10.5 |
Application Server |
Websphere 7.0.0.35 |
Websphere 8.5.5 |
Operating System |
Redhat Enterprise Linux (RHEL) 5 |
Redhat Enterprise Linux (RHEL) 7 |
*
Unsupported on CLM 5.0.2
The recommended order for the situation above would be:
- Upgrade the CLM Operating System(s)
- Upgrade the Application Server
- Upgrade the Database Management System
- Upgrade CLM
Review your topology
When planning an upgrade, review your topology to ensure it will meet your business needs. This is the time to consider changing it if necessary.
To simplify CLM deployment options, IBM has outlined several
topologies which are the recommended and most frequently chosen deployment patterns seen in the field. They represent tried and true examples of how customers have successfully deployed the CLM solution and are used by our system testing organization to perform deep “customer simulation” testing which includes installation, upgrade, functional, performance, and robustness testing.
When the need to switch topologies, application servers, databases, or other resources is discovered, make sure to migrate only one system part at a time where possible to isolate changes to the system.
Verify that your hot fixes are in the release
To avoid regression, an important step in upgrade planning is to ensure that any hot fixes that have been deployed in your enterprise are fixed in the new release. Where there are gaps, contact IBM Technical Support who will help you get the necessary fixes.
You can get a list of hot fixes by clicking on
about this application in the top right-hand side of the web UI. For Rational Requirements Composer releases prior to 5.0, you will need to maintain a separate inventory of hot fixes that have been applied.
Testing
Setting up a staging environment
Ideally, a staging environment is a test sandbox that includes a snapshot of production data isolated from the production environment.
It is
imperative that you test the upgrade procedure prior to attempting it with your production system.
It is strongly recommended you do this with your
production data, so that you can fully test links and artifacts.
See
Topologies and mapping files for setting up a test staging environment for details.
Finally, the test environment should be ‘clean’ and not used for other evaluation or testing purposes unless the changes can be completely backed out.
The options, at a glance, for achieving a staging area with production data are:
- Isolated subnet environment
- Isolated Virtual environment
- Use a local hosts file
- SOCKs server
- DNS server
Reasons for a staging environment
The main three good reasons to test an upgrade are:
The planning process and documentation to be used during the upgrade depends upon the timings and bespoke steps ascertained during testing.
You are also looking to:
- Define a repeatable process
- Realize any defects now
- Run any performance tests
- Experimentation; particularly for users to understand new features
You should be or become very familiar with the
monitoring section of this wiki during testing, to be able to recognize performance patterns during upgrade. Knowing when and whether your system should be under stress could be the difference between a successful, or aborted upgrade.
Further considerations for testing
There are several techniques designed to ensure that the testing of the upgrade, through iterations, as well as the upgrade itself run more smoothly.
Although production data is preferred, there are two recommended scenarios ypu can use to validate your CLM upgrade:
There is also the historical project:
- CLM Build Verification Test (BVT) - This test scenario, available on jazz.net, is based on a CD Classic application that provides a site to buy and rate classic music CDs.
Downtime Mitigation
Below are some of the strategies used by customers to mitigate downtime during an upgrade:
- Collocation - Temporary moving the DBMS closer (or in some cases on) to the application server
- Temporarily increasing system resources (CPU/RAM) and increasing the heap size for repotools
- Keeping the machines use exclusive during the upgrade in the case other applications are running any consuming system resources
- Cleaning up any unwanted data prior to the upgrade. For example, large sandbox projects or even instances of the application that have been migrated and no longer used
- Re-indexing applications prior to upgrading to ensure index integrity (repotools- -reindex all). See repotools command reference for usage.
Utilizing experiences
Rollback and recovery
The IUG provides guidance in the event you need to back out of an upgrade. Make sure you have a back out plan.
Built on top of the IUG, our client facing teams have developed good practice tips based on their experiences upgrading the CLM application suite, which is summarized below and
- Restore points prevent abandoned upgrades
- Document a full rollback procedure
- Don’t panic – it’s a tested part of the procedure
As with any backup and recovery procedure, it is important that the back out plan is tested. It is likely that you will only need to return to a previous restore point, for example, restore to the point of having successfully upgraded the Jazz Team Server. You may need to coordinate a virtual machine rollback to the point of a database backup, so ensure these tasks are completed concurrently.
Upgrades are successful when...
We collated experiences of the successful upgrades, even among the most complex of scenarios. Upgrades are successful when...
- Defined custom test cases for common usage scenarios
- Detailed Use Cases define how to perform actions
- Step-by-step plans for overall upgrade
- Full detailed plan of the day/weekend upgrade, including who, when, how etc..
Trial runs of upgrade
- Append newly learned information to plans and improve upgrade procedures
- Provide better estimates of each action and more accurately predict downtime for end users
- Catch problems before the day of upgrade
Reporting considerations
Over time, there has been updates to the recommended upgrade order when there exists a reporting server (either RRDI or Insight) in your deployment. It is recommended that you review the
Version Compatibility Matrix for CLM, RRDI, and Insight prior to upgrading. Currently:
- CLM should be upgraded before RRDI/Insight
- It is not required to upgrade CLM/RRDI at the same time
- Follow pre-upgrade tasks for upgrading RRDI (disabling SSL and backing up configuration)
- If using Insight, skip the data warehouse upgrade during the CLM upgrade and run migrateDW first. Then run the repotools-jts –upgradeWarehouse
- Like other applications, DCC should always be at the same level as the JTS
It is worth reiterating the benefits that were mainly introduced in 5.0 for consideration:
- Enable the Jazz Reporting Service (JRS) (5.x)
- Improve dashboard performance by replacing widgets with JRS reports when possible
- Allow the average user to create lifecycle traceability reports
Enable the Data Collection Component (DCC) (5.0.x)
- Disable Java ETL or Data Manager CLM ETL (Insight)
- Delegating ETL processing to the DCC takes the strain of ETL processing away from the CLM systems. Great for globally distributed deployments.
- Provides faster and more frequent ETL refresh for closer to real-time reporting data
Evaluate using Online Migration
As of 4.0.6, online migration and estimating is available for RQM(Not available for 5.x to 6.x upgrade, see Note below). As of 5.0, it is available for RTC as well. Offline migration is still the recommended path as it allows for easier recovery from migration errors than an online migration.
Before implementing an online migration, determine whether offline or online migration is warranted for your repository by running the migration estimation command and then checking the results against the following test matrices:
RTC Online Migration Test Matrix
RQM Online Migration Test Matrix
Repository tools command to evaluate the online migration process
*Note:* There are some important updates regarding Online Migration noted on the Upgrade Insider page.
Final checklist
As this article explains, the majority of work for an upgrade is prior to the actual event itself. The last appreciation of an upgrade is that it is usually performed by real people, over a weekend and/or late at night. Therefore, reducing the manual steps and ensuring all eventualities are documented will relieve pressure on those executing the upgrade.
Here is a basic checklist for the upgrade event itself and a summary of this article:
- Staged communications
- Don’t deviate from the plan
- Ensure operator follows each step
- Abort and return another day
- Track each step against experience
- Monitor for deviance from expected
- Once complete run confirmation tests
- Scour logs to ensure error-free
- Inform users system is available
Contact us when you need help
There are several ways in which you can get help:
You can contact your Technical Sales Representative or, if you are an Accelerated Value Partner, then you can also reach out to your Accelerated Value Lead to review and answer any questions you have regarding the upgrade.
If you would like Technical Support (see
Contact us) to evaluate your plan and provide feedback and recommendations, please open a PMR via the
Service Requests & PMRs tab on the
Support Portal.
Provide as much detail about your environment as possible and allow sufficient lead time for evaluation (especially if the upgrade is planned during the weekend).
You can also post questions in the
Jazz Forum on jazz.net. It is monitored by the jazz community and email notifications let you know when your question has been answered or has been commented upon. Before posting your question, please search the forum as it might already be answered.
External links: