I am one of the developers in the Rational Collaborative Lifecycle Management (CLM) project contributing to the Tracking and Planning (TAP) functionality of Rational Team Concert (RTC). I have been part of the TAP team from last four years and involved in delivering RTC in longer and shorter release cycles. You would have read posts written by my colleagues on how we changed our processes and organized our teams for adapting to the Continuous Delivery lifecycle. This blog is the continuation to these posts, in which I’ll be talking about the “Run Team” and its necessity and role in delivering quality values to our product. We have been running the Run Team from the last four releases (since 4.0.1) and I was part of the team for three releases and led the team during two releases.
Rupa’s recent post on how we organize our teams describes the difference between Feature Teams and Run Teams. Run Teams are a recognition that not all of the development work in a release of RTC is work on new features and enhancements. There is also significant work which happens around bug fixing, answering users questions on the Jazz.net forum and working on a backlog of tasks and improvements not directly related to the new features we want to deliver in the release. The Run Team’s job is to handle this type of work and leave the Feature Teams free to focus on their new features. The Run Team is a rotational assignment which gives its members an opportunity dig deeply into all areas of our product and continuously increase their technical skills. The Run Team keeps the day-to-day operations of the release running smoothly, leaving the Feature Teams free to give their plan items all of the focus and attention they require.
Responsibilities we defined for the Run Team:
The TAP team is responsible for the change management (work items) and planning (plans) capabilities of RTC. We have the following responsibilities defined for every release:
- Implementing features and enhancements planned for the release
- Improving the quality of TAP component by fixing defects and regressions
- Constantly interacting with the Level 3 Support Team (L3) to collaborate on customer escalations and fixes for defects submitted through official support channels – we call these Authorized Problem Analysis Reports (APARs)
- Continuous builds and deployments. Monitoring builds, push & pull changes to integration stream.
- Tracking & planning defect backlog and triaging new incoming requests (Inbox monitoring).
Earlier, when there were no Run Teams and Feature Teams, the entire TAP team used to take care of most of the responsibilities. After adopting our Continuous Delivery process and forming a Run Team, we assigned majority of these duties to it. We came up with three roles and defined the process and responsibilities to be performed by each.
- APARMeister: Role for single point of contact, responsible for interacting with L3 team on customer escalations for decision making on APARs and Requests for Feature Enhancement (RFEs) submitted through official support channels.
- BuildMeister: Role responsible for monitoring builds, maintaining continuous and integration streams on same baselines and deployment of builds on our test bed.
- InboxMeister: Role to go through the TAP Inbox and do triaging, prioritizing and planning work for current milestone or release or future release or backlog.
In addition to the above roles, the complete Run Team has the additional high priority task of fixing defects, regressions and APARs to improve the stability and quality of the product.
How do we work in Run Team:
Since the Run team is a separate independent entity which has its own duties, goals and targets, we have created a new team area for TAP in RTC called Run Team. We associated all our existing categories under planning and work items to this team area which let the Run team to own all the work that is filed against these categories. For the features work we created new categories and associated to the feature team. Teams working on features during a release should implement them completely. From next releases, these feature categories are owned by the Run Team. Any work items created under these categories are either defects or additional enhancements (which are not in scope while implementing this feature) and are monitored and worked by the Run Team.
We leveraged RTC plans and work items for planning and tracking the Run Team’s work. For every milestone, the Run Team lead creates a plan for the team and organizes the plan’s dashboard with different queues for APARs, regressions, Severity1/Severity2 defects and backlogs. The Run Team lead uses these queues for planning work to the current milestone or release. He/she also defines the rules and targets for the team and meisters for the current milestone in the plan. We used pull model for assigning work to the team. The Run Team Plan shows planned, prioritized (ranked) and unassigned work under Unassigned group. Team members are free to pull work for themselves from this unassigned group starting with high priority ones. Please refer to Run Team Dashboard and TAP Run Team 4.0.4 M1 Plan for example.
Following agile methodologies, the Run Team lead runs the daily scrums and acts as Scrum master for the scrums. These are short meetings which do not span for more than 15 to 30 minutes. We do a status round robin where the team updates about their assigned work and also discusses on any blocking issues. Meisters provide their update to the team with any build issues, escalations, APARs, new incoming defects or enhancements. The Run Team lead takes the inputs from the team and plan and prioritize these items based on team’s targets and load. We followed a round robin approach for Meister role assignments by rotating each role across the team to ensure that every team member performs every role during the release.
What are the Outcomes we observed:
- Team acquired knowledge and expertise by working/fixing defects across all areas.
- Feature teams are shielded from most of the external duties thereby focussing only on implementing plan items end to end.
- Run team focused on achieving its primary objectives – meeting targets for fixing APARs and other quality goals. This results to the release with expected quality.
- Having a separate role for APARMeister helped to improve the interaction with the support team and controlling the APARs flow.
- Maintains continuous builds and deployments.
- Controlled inbox and backlog.
- Daily scrums helped the geographically distributed team to be on the same page and consistently work towards the defined targets.
- Every team member gets experience on handling and managing builds, escalations, inbox and related issues by following rotation policy for meister roles. It also balanced the work load on the team.
How have you organized your teams to support continuous releases? Please feel free to comment below.
Staff Software Engineer, Tracking and Planning Team,
Rational Team Concert, IBM Rational