From Waterfall to Agile with the Rational solution for Collaborative Lifecycle Management
How does a software development team become “Agile” when they develop a product line that is used by over 90% of IBM’s top 500 customers, that has been identified as one of IBM’s 100 icons of progress, and that has been developed for over 40 years in assembler PL/X and COBOL, but now has Java and XML thrown in?
Can a product with as much history and success as CICS adopt IBM’s Rational solution for Collaborative Lifecycle Management (CLM) and continue to be successful? Of course it can!
This is a story of the seven year journey for the product team developing IBM CICS Transaction Server for z/OS (CICS TS for z/OS), and its family of CICS Tools and connectors. CICS TS for z/OS provides industrial-strength, online transaction management and connectivity for mission-critical applications.
Illustration 1: IBM Hursley Laboratory, UK – Th home of CICS
The journey started in 2005, with the recognition that the adoption of Agile principles rather than the existing waterfall process would benefit the product development cycle Using the waterfall process, defects could peak at over 600 and, with integration testing done so late in the development cycle, the most complex problems would be found late on. With a beta shipped only months before availability, it was unlikely to get any customer feedback into the product in time for the release.
So the team made the strategic decision to become more agile. But how to do this when there were few skills and little knowledge in the team of Agile development, and they still had a two year release cycle and no well travelled path to follow? The team were not totally convinced of the benefit of this new way of working.
The first steps were to break the development process into four-month iterations, with coding, unit testing, functional testing, and documentation all being done together. A beta was released at the end of each iteration, and integration or system testing would take place in the following iteration.
The cultural change of 40 years following waterfall development, the large code base (Millions of lines of code) written without thought of Agile methodology, along with large requirements and Agile principles still being new to the team, meant that the change was exceptionally challenging.
After two years of development, and the first release using this approach, the team did see a number of benefits. Integration testing was being done much earlier, and defects (particularly complex integration ones) were being discovered much earlier, leading to improved product quality. Importantly, the first experience of Agile development changed people’s perception of what was possible. Work could be broken down into smaller ‘chunks’.
Fast forward to 2008, and the IBM CICS team perform a development retrospective. There were real concerns over:
- Multiple source code management systems, some of which dated back to the 1980s, which had been extended, configured and specialised for CICS use
- A unique, specialised set of build engines which were tightly coupled to the library systems in which the skill sets were diminishing
- Development and service teams working in completely different environments.
- Disparate tools for requirements, designs, documents, etc.
- Large management overhead in planning, tracking, recording status and managing change
- Teams working inconsistently with respect to documentation and process
- Very high learning curve for new starters
Based on these concerns, the CICS TS for z/OS team set a vision of a “single non-proprietary environment for the delivery and service of future releases of CICS TS for z/OS, where everyone (business, marketing, development, test, service, build, etc.) can focus and collaborate via one single tool.”
So why RTC?
In 2008, a recently released IBM Rational Team Concert (RTC) V1 beta was seen by a couple of CICS system testers, and they decided to use it for their test infrastructure. Their experience led to RTC being tested by several other CICS team members, and these early trials showed great promise. In 2009 the IBM Hursley Lab started centrally hosting Jazz servers, enabling the team to request a server and set up their initial projects within days.
RTC proved to be highly configurable and appeared to give the team the end-to-end integrated development environment that they were looking for. Requirements, approvals, defects, code, tracking, and build – and all with associated history for all changes, enabling detailed audit tracking.
With the features provided in RTC, such as work items, teams, categories and time lines, we saw that it offered great support for Agile development.
In June 2009, following a major release of CICS TS for z/OS, the team adopted two-month iterations, still with four-month betas, and work started on the migration to RTC. It was recognised that it would take some time to get to the complete RTC solution, due to having to rewrite the build to separate it from the current library systems.
The initial focus was going to be on work items, project planning and tracking; such things as Epics, Stories. Tasks, Defects, Risks, Actions, etc. Migrating the majority of the source code to RTC, rewriting the build using Ant, and providing all the tooling, infrastructure and education for developing the code in RTC was going to take much longer.
One of the advantages of RTC was the ability to adopt functionality in an ongoing way. If the team had waited for it to be perfect, they’d have ended up never using it!
The transition to RTC was staged with the aim being to use RTC ‘out of the box’ and to try and have no proprietary code if at all possible.
Planning and Track adoption
Adoption of RTC for planning and tracking purposes began in July 2009:
- July 2009: Created RTC server
- Created an infrastructure project to use for the planning and tracking of the migration to RTC
- Created a main project for the development of CICS TS for z/OS
- Created a sandpit project that was effectively a duplicate of the main development one and was used for trying out changes to make sure they worked
- August: October 2009 – Reviewed processes and configured projects – 80-90% right
- Mid-2009: migration of Epics (requirements) to RTC
- October 2009: migration of Defects from existing defect management tool to RTC
- November 2009: migration of HFS based source code to RTC
- End of 2009: using RTC for all work outside of propriety source/build environment
Build adoption
With the CICS team’s direction being to remove the need for proprietary tools, and with the skills diminishing on the existing build tools, the decision was taken to rewrite the CICS TS for z/OS build from scratch.
At the time CICS TS for z/OS started to adopted RTC, its build capability did not provide the features they required. CICS has some very specific build needs and technologies and so the decision was taken to rewrite it based on Ant technology.
- January 2010: Start of build rewrite using Ant
- September 2010: First integration build (used for integration testing)
- October 2010 (limited release) / February 2011: First beta shipped from the new Ant based build engine
- June 24, 2011: RTC delivered CICS TS for z/OS V4.2
Delivery and Test Services (DTS) tool adoption
The CICS TS for z/OS team were using a tool that handle automated testing and code promotion. At the time CICS TS for z/OS started to adopt RTC, this function was not available in RTC, but was felt to be critical to the environment that the team worked in.
DTS, which has been written by the CICS TS for z/OS team, replaces the the previous code promotion tool. Builds for CICS TS for z/OS can take many hours to complete and, due to the length of time to run, are usually run overnight. DTS has been written to automate many of the tests and automatically promote the code should the tests pass.
The team have a number of streams that are used for managing the delivery of code:
- Integrated stream: This is the stream that the developers submit their changes to after they have unit tested the code
- Best so far: An intermediate stream, to which code is promoted when it has passed an initial set of automated regression tests
- Increment: The main development stream which is used for testing, and from which the product is shipped. Code is promoted to this stream only when further, more detailed, automated regression tests have been run
Without the streamed approach and the use of DTS, it was far more likely that damaging code could make it into the product, and could break it. This could stop the team from testing for a number of hours and, as has occasionally happened, days.
- June 2010: DTS started
- March 2011: DTS ready
Development environment
The goal was for the new development and build environment to be Eclipse based and consist of:
- Rational Team Concert v3
- Rational Developer for System Z v8
- IBM Internal language editor plug-in for better context aware editing
- Ant build technology as an Eclipse plug-in for build control
The investigation started in parallel with the rewrite of the build and the development of DTS:
- April 2011: Developer environment ready
- June 2011: Full Source Code Environment switched to RTC
- June/July 2011: RTC/RDz development environment rolled out
Following the delivery of CICS Transaction Server for z/OS Version 4.2 in June 2011, the team transitioned to this new environment.
Illustration 2: The new development environment since June 2011
The benefits…
So has the adoption of Agile principles and Rational tools realised any benefits? The simple answer is yes, but often it is not as clear cut as being able to say that x amount of time has been saved or x amount of money has been saved, although clearly it has. It is more about providing an integrated, consistent and open environment where everyone can collaborate together.
These are some of the direct benefits that the team is seeing:
- Integrated work item and reporting tools
- Transparency of work items, dependencies and status – instant status reporting through dashboards
- Ability to ignore what is on track and focus on the issues
- Risks/Actions integrated and linking to the work items they impact
- More efficient status meetings:
- Preparation not required as dashboards are used for live information – previously presentations were used that needed creating/updating each week
- RTC provides up-to-date information
- Meeting times reduced by a third
- More efficient end of iteration meetings:
- Preparation time reduced by 75%
- Meeting times halved
- Responsiveness to business needs – aiming for greater value, earlier
- Beta deliveries much earlier
- Ability to adapt to change much easier
- Flexible resource pool, with common skills
- Ability to handle requirements and prioritisation in a much more agile way
- Quality has improved
- Defects in RTC and so part of the backlog
- Pre-2005 defects peaked at 600+ in waterfall development
- 2005-2009 defects focused on much earlier (450 peak) – Agile/Iterative
- 2009-2011 defects peaked at even lower level (350 peak) – Agile/Iterative+RTC
- Far greater ability to use Agile practices. Most teams are now working in one-month iterations. This could not have been imagined two years ago
- Far greater collaboration across teams
- Far greater notification when things change
- New build requires two people to support it, rather than three for the previous build
- Using an Ant-based build engine, the need for specialised CICS build skills is reduced
- Additional benefits compared with the previous development environment:
- Skills in ‘industry leading’ tooling
- Eclipse or Web interface
- Easier to move roles within CICS or Hursley
- The development and service teams are now working in an identical environment, so it is easier to transition roles
- Single source code management
- Far easier learning curve for new joiners, particularly as many graduates have experience of the Eclipse development environment.
- After 3 months of RTC/RDz, 90% of developers are working as well or better than the previous environment.
To quote Simon Rachman, the CICS TS for z/OS Delivery Manager, “RTC brings a huge amount of added value to managing projects. Its power, when used well, is that it enables a complete view of project workloads and progress, all with less effort. Having the right information to hand means you are so much more in control and can focus extra effort exactly where it is needed. These factors are essential if you are going to avoid the extensive additional costs incurred by projects that run late.”
Some of the benefits are not so easy to bullet point. For example, during waterfall development when developing large changes, it is difficult to accurately size and almost impossible to de-commit should the work overrun. This often meant that the team could be working for a number of months at the end of the release completing delivery of a function. Since the adoption of Agile practices and certainly RTC, there is much more of a sense of being in control of the work being committed and delivered, and for CICS Transaction Server for z/OS Version 4.2 there was no overrun at all.
Where is the team now?
Since the adoption of the new development environment in June 2011, the CICS team has not stood still. A number of initiatives have needed to be taken forward.
Team reorganization
At the start of a new release of CICS TS for z/OS, the management team reorganised the teams to best suit that project. This meant updating RTC to reflect those new teams. Making the changes to support this task in RTC took less than a day and was limited to creating the new teams, deleting the old ones, assigning the categories to the new teams, and setting up the dashboards to reflect the changes.
Access control and compartmentalisation
In order to better adhere to IBM standards around access to information and source code, the team decided to separate the source code into separate projects. While RTC does provide levels of access control, this was a simple and effective approach to the problem.
Further adoption of Agile practices
The team have also invested time in the further adoption of Agile practices:
- Most teams have moved to one-month iterations
- More regular stakeholder feedback is provided with a managed beta program every two months and internal stakeholder reviews on a regular basis
- Teams are now finding it easier to adopt Story point for sizing
- Use case definitions (actor, goal, value) are provided to aid in defining what is expected in the product
- Iteration planning is priority based with team driven commitment
New customer requirements solution
The CICS portfolio of products has adopted developerWorks Request for Enhancement (RFE) for customers to raise requirements. The CICS requirements are linked directly into an RTC project on the CICS Jazz server. For the first time, this allows customer requirements and their requests to be linked directly to the products the teams are developing.
Further deployment of Rational Quality Manager (RQM)
While the primary focus has been on the adoption of RTC, RQM has also been used. Initially it has been used just for test execution tracking, but the intention is to look at how it might now integrate with the test tooling the team are using and see if any automation is possible.
Portfolio planning using Rational Requirements Composer (RRC)
RRC allowed the team to identify that they now had the tooling necessary to look at the planning and handling of requirements for all the products in the CICS portfolio. This process is currently being prototyped, but allows the strategy and planning team to define the vision, roadmaps and features for the products. The customer requirements from RFE feed directly into these artefacts, and from the features and the use cases in RRC the Epics and Stories will be created in the product RTC projects. This results in a very tidy way for managing Epics and Stories for the release currently under development (in RTC) and those planned for future releases (in RRC).
The last release of the Jazz product provided the ability to create Collaborative Lifecycle Management (CLM) projects. This allows all of the project types (RRC/RTC/RQM) to be pulled together, and provides traceability links that show how Features are implemented by Epics/Stories and validated by test plans. It can even list the defects associated with that requirement. Effectively, everything can be traced from the initial requirement to the developed code.
The diagram below shows the outline architecture that has been created for the development and project management of the CICS portfolio of products. It shows not only how the projects connect, but also which ones are used primarily for planning and which for delivery.
Illustration 3: Overview of the CICS Jazz server architecture
Summary
Even for a product as mature and critical as CICS TS for z/OS, the transition from a waterfall-based approach to one based on Agile principles can clearly be achieved. It takes time and determination, and the right tooling is essential. RTC has provided the CICS team with this integrated tooling.
RTC and Agile methodology have allowed the team to adopt the various practices and tools in a staged manner and significant benefits are now being realised. With the continued adoption of RRC and RQM and a full CLM solution, further benefits will continue to be realised in the future.
IBM Hursley Laboratory
In addition to CICS TS for z/OS, the adoption of Rational tooling has expanded to many other teams at the Hursley Laboratory. These include:
- Websphere MQ wanted to move to a more agile development process using Rational Team Concert
- Websphere MQ and WebSphere Message Broker Teams wanted a better way to manage and track test activity using Rational Quality Manager
- and more
Since 2008, the growth in Rational tools as part of a centrally supported, single Agile development solution at IBM Hursley has been staggering. The team that hosts the Jazz servers initially deployed Jazz on what was at the time an unsupported platform (AIX), as that was the only equipment that was available at the time. AIX is now supported, and the team have had to put in place capital plans to support the existing and future anticipated growth they are seeing.
 Chart 1: The number of Jazz repositories hosted at Hursley |  Chart 2: The number of users on Hursley Jazz repositories (000’s) |
 Chart 3: The size (Terabytes) of the file systems on Hursley Jazz repositories showing the aniticpated growth for the remainder of the year |  Chart 4: The distribution by country of Hursley Jazz users |
Until 2010, the adoption of Jazz at Hursley was not driven by management, but by the users themselves. It was only when the management team found that 80% of the development teams in Hursley were already using Jazz tooling, or planning to use it, that the message was given that all new projects were to use Jazz as the basis for their development.
In 2009, the growth was so great that the Hursley Jazz Community was formed. It meets on a monthly basis to share best practices and help new adopters overcome any problems they might be experiencing.
The team supporting the Jazz servers have been surprised by the eagerness of users to adopt the latest version of the Jazz product. In many cases requests being received for the next version before it has been released! Previously the team have found it extremely difficult to pry users off existing systems.
This has been a fascinating challenge for the Rational Tools team at Hursley, and will continue to be so as they support hundreds of servers and tens of thousands of users into the future.
Copyright © 2012 IBM Corporation