Jazz Library Jazz Administration Guide
Author name

workflow-management,rational-team-concert

Thu, 17 Jun 2010
5 min read
0

Jazz Administration Guide (2.x)

The Jazz Jumpstart Team: Dan Toczala, Technical Focal Point
Last updated: December 13, 2010
Build basis: Rational Team Concert 2.0.0.2

Disclaimer: This guide is an evolving document and incremental improvements will be published on a regular basis. Feedback is welcome. Please direct your comments, questions, and suggestions to Dan Toczala (dtoczala@us.ibm.com)  The contents of this guide reflect the current Best Practices for Jazz Administration, as collected by Rational field teams.  Much of the base content of this document comes from materials and experiences harvested from the Rational Engineering Servcies Jazz Operations Team.  Their contribution to this has been quite valuable. 

This is an example that can be used as a template for a Jazz/RTC Administration Guide.  This is not meant as a complete product, but should serve as a template for the creation of Jazz/RTC Administration Guide that should be used to help provide a consistent level of service for the end users of a Jazz/RTC deployment.  Many of the sections require the input of installation and site specific information.  This is designed to be tailorable, and you will probably add and remove entire sections of this document, in order to improve the value of what is here, or to meet the documentation requirements of your organization.

You should be able to do a simple cut and paste starting from the title below to get the content that will serve as the template which you can then fill in.

Note: A later version of this guide is available for the RTC 3.0 release.

Administration Plan for Jazz Platform

Prepared by: Your name here
Last Updated on: June 6, 2010

Table of Contents

Introduction

This document defines the set of tasks performed by the Rational Team Concert (RTC) administrator to keep the environment up and running effectively. It provides recommended guidelines in each administrative area. The implementation details can be found on Jazz.net, the product help files, or in the various IBM hosted forums and informational pages found on developerWorks. Some of these responsibilities are periodic; others are performed as required. The activities include:

  • Setting up the Jazz/RTC servers
  • Maintaining and monitoring integrations
  • Backing up and restoring Jazz/RTC databases
  • Periodically maintaining the data repository (archiving old data)
  • Setting up, monitoring, and managing license usage
  • Monitoring and tuning performance
  • Installing patches and update areas
A general review of the activities and issues encountered with deployment of Jazz based products can be found in Getting Started with Rational Team Concert: A Deployment Guide.

Architecture Overview

There are three elements to consider in the RTC environment: setting up the web server, setting up the database server, and optionally establishing the required integrations. These procedures are followed when doing the initial setup of a RTC Server environment. This section describes the procedures developed in each area.

RTC Server

The RTC server is a Java-based web application that runs within any Java EE 1.4 compliant application server. Two different application servers are supported in the 2.0 release of RTC: Apache Tomcat and IBM WebSphere Apache. The semantics of services and server-side RTC APIs are independent of the choice of application server, allowing additional application servers to be supported in future releases.

Supported Application ServersDeployed
Apache Tomcat 5.5.23 
WAS 6.1.0.23 with Fixpack 85942
IBM WebSphere Application Server 7.0.0.3		 

Database Server

The repository is back-ended by a relational database. The 2.0 release of RTC supports three relational database systems: Apache Derby, Microsoft SQL server, IBM DB2 , and Oracle Database.


Supported Database ServersDeployed
IBM DB2 9.5 
Oracle Database 10g release 2 
Apache Derby 10.3 
Microsoft SQL Server 2008 or 2005 SP2 

Authentication Mechanism

RTC relies on the application server’s standard LDAP-based authentication mechanisms for login and identity checking. Once past these mandatory checks, the RTC server maps the identify of the logged-in user to a RTC contributor item which identifies the user within RTC and serves as the basis for further permission checks. This two-tired approach makes it easy for an organization to control who gets access to a RTC server while delegating finer-grained authorization decisions to the project leadership to control from within RTC.

Supported LDAP Servers for Identity ManagementDeployed
Apache Tomcat 5.5.23 
IBM Tivoli Directory Server 6.0 
Microsoft Windows Server 2003 Active Directory 

In order to add thousands of Dashboard users a number of existing LDAP groups must be mapped to the JazzGuest group. On a nightly basis, RTC will bring those users over into RTC.

Note: The RTC Team Server user identity is case sensitive. When using LDAP for user management, turn off the case-insensitive option. Work with your server administrator or consult your product documentation to ensure that the settings are case-sensitive.

RTC Client Interface

RTC provides users with both an Eclipse-based client interface and a Web interface. The client interface provides developers with a rich, integrated development environment for building and delivering artifacts. The RTC web UI is better suited for casual or occasional users than the IDE because it does not require installing any special software on the client machine; all that’s needed is a web browser.
For Visual Studio users, a plugin for the Visual Studio IDE is provided which will allow users to access Jazz/RTC functionality from within the Visual Studio IDE.

Supported Client InterfacesDeployed
Eclipse Based Client 
Web Interface 
Microsoft Visual Studio .NET Client 

The client installation instructions can be found online, as well as a list of the required Eclipse versions, and supported web browsers.  Note that Internet Explorer 6 is NOT a supported browser.

Integrations

Currently there are a limited number of products with integrations to the Jazz infrastructure, but the number of applications and tools that interface with the RTC infrastructure are expanding as time goes on. The integrations can be broken down into two categories: connectors and bridges.  the list below is not comprehensive, as there are a large number of other integrations available.  For a full list of vendors providing integrations and Jazz friendly applications, see the IBM Rational Ensemble.

Common Connectors and BridgesDeployed
ClearCase Connector 
ClearCase Bridge 
ClearQuest Connector 
ClearQuest Bridge 

Integration X Information

Here you would include information on your specific integration, as well as any specific configuration details needed to support the integration.  If you do not need this section, then please remove it.

Hardware/Software Requirements

Here you would include information on your specific hardware and software needs.  If you do not need this section, then please remove it.

Installation Information

 

Here you would include information on the installation of your specific integration.  If you do not need this section, then please remove it.

Configuration Information

Here you would include information on the configuration details needed to support the integration.  If you do not need this section, then please remove it.

ClearQuest Connector Information

Note: This information is supplied as a sample for you to use as a basis for your own information. If you do not need this section, then please remove it.
The ClearQuest Defect records will be synchronized with the Defect work items in RTC. The tool of record is ClearQuest. Defects are input into ClearQuest and then synchronized with RTC. Once in RTC, these defects are synchronized back with the original source. Defect records input into RTC will not be synchronized. This synchronization is a bidirectional synchronization, however a non tool enforced business rule has been put in place that all Defects will originate in the “System of Record”. The “System of Record” is ClearQuest.

The CQ Connector uses a server process, the ClearQuest Gateway, to establish communication between a CQ database and the RTC server. The ClearQuest Gateway can be connected to only one CQ user database.

Hardware / Software Requirements

The CQ Connector is a tomcat instance running on a Windows or Linux machine.

ComponentVersion or Configuration
RTC Server Software2.0.0.2
RTC Server HardwareRHEL WS4 (http://rtcserver.myorg.com)
ClearQuest Connector Software2.0.0.2
ClearQuest Connector HardwareWindows 2003 Server (http://sample.myorg.com)
ClearQuest Client Software7.0.1.2

Installation Information

The download of the CQ Connector is located in a download from https://jazz.net/downloads/rational-team-concert/. All of the installation information is contained within the zip file. There are no other pieces to install for the CQ Connector.
  1. Download the ClearQuest Connector .zip file.
  2. Extract the .zip file into the JazzInstallDir directory.
  3. After the installation is complete, configure the ClearQuest Connector. See Deploying ClearQuest Connector.

Configuration Information

To configure the CQ Connector with the Setup Wizard, the Jazz Team Server and ClearQuest Gateway must be running. The ClearQuest Gateway must be running on a computer running Windows. The wizard generates a properties file, cqconnector.properties, that the ClearQuest Gateway uses.  Be sure to read the documentation on Configuring the ClearQuest Connector.

Configuring the Synchronizer
Read and understand the documentation on Configuring the ClearQuest Connector.  Some key points to keep in mind as you do this:
  • Note the name and password of your ClearQuest user account in RTC (often it is cqconnector).
  • Have a specific login name and password for logging into ClearQuest for synchronization operations.  The user account must have SQL editor privledges.
  • Conector queries need to be public queries.
Configuration of the XML synchronization rules
This publication will guide a user through the setup of the initial synchronization rule. However, there should not be any need to create a new rule from scratch. http://publib.boulder.ibm.com/infocenter/rtc/v2r0m0/index.jsp?topic=/com.ibm.team.connector.cq.doc/topics/c_prerequisites.html
ClearQuest FieldMappingRTC Work item Attribute (field)Notes
ID>CQ_IDOriginal ClearQuest record ID
Attachments<>Attachments 
State<>StatusRTC work item states are a subset of the ClearQuest lifecycle.
etc.etc.etc.etc.

Monitoring
After the creation of the synchronization rules is complete, the administrator and team members should monitor the status of the rule(s). This task should take no longer 30 minutes.
Synchronization StatusDescription
ConflictConflicts can occur when two users make nearly simultaneous changes to a ClearQuest record and its corresponding work item. The changes result in a ClearQuest field and the work item property to which it is mapped containing different values. You need to resolve the conflict so that the work item property and ClearQuest field contain the same field values.
CycleToo many incoming and outgoing synchronization operations have occurred without completing successfully.
Incoming errorAn error occurred that prevented incoming synchronization of changes from ClearQuest from completing successfully.
OKAn incoming-outgoing or outgoing-incoming cycle has occurred and the resulting states (with respect to the properties being synchronized) are equivalent.
Outgoing errorAn error occurred that prevented outgoing synchronization of changes from Rational Team Concert to ClearQuest from completing successfully.
Pending (blocked)Changes could not be applied due to a dependency on another object that is not yet connected.
Pending IncomingOutgoing changes (from Rational Team Concert to ClearQuest) have been applied. If the next incoming state is equivalent, the status changes to OK.
Pending OutgoingIncoming changes (from ClearQuest to Rational Team Concert) have been applied. If the next outgoing state is equivalent, the status changes to OK. Uninitialized This is the initial state. Synchronization has not occurred.

Installation and Configuration

Software Installation and Configuration

The following table provides a general overview of the installation and configuration procedure.  For more complete instructions on installation and configuration, see the RTC Installation Instructions.

PhaseDescription
Plan InstallationMake decisions on the architecture and collect information that will be required during the installation and configuration of the environment. Refer to the Installation Configuration Documentation Worksheet reference.
Verify Database Server Pre-RequisitesDatabase software is installed and running on a machine to be used as the database server. In production environments, this machine should be a different one from that on which the RTC Team Server runs.  The test and evaluation environments this may reside on the same server.
Verify Application Server Pre-RequisitesApache Tomcat should be properly configured when used as installed with RTC.

WebSphere Application server software is already installed. Verify authentication case-insensitive property is turned off.
NOTE: Although WebSphere and many LDAP directories allow you to log in case-insensitive, RTC Team Server stores user records with user IDs in exact case as they are imported. When you log in to the RTC Team Server, the user record is retrieved from the list of users and the case must match exactly.

Verify that Java™ 2 Security option is turned off.
NOTE: If this option is turned on in WebSphere Application Server it will cause the RTC Team Server .war to fail to start.

Verify or update the WebSphere Application Server level.
NOTE: RTC Team Server requires WebSphere Application Server Version 6.1 with the IBM Java SDK 1.5 SR5 or later Cumulative Fix applied. The IBM Java SDK 1.5 SR6 SDK update is available from http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg24017492. With WebSphere 6.1 and the GA IBM Java SDK, RTC starts to report “proxy errors” after some number of operations.

Please update any site specific settings or important notes in the Installation Configuration Documentation Worksheet.
Obtain Installation FilesDownload RTC Team Server and Rational Team from Jazz.net.
NOTE: Download files to be installed with Installation Manager.  You can also do a zip install if you choose to do so.
Install the RTC ServerFirst read through all of the step-by-step installation instructions, then you can go through them and execute them

Please update any site specific settings or important notes in the Installation Configuration Documentation Worksheet.
Set Up Database ServerIf you are doing this in a demonstration environment, then you can just use a Derby database, since these are easy to setup and small.

If you are doing this for a testing or production environment, you will need to follow the specific directions for the DB2 database setup, the Oracle database setup, or the SQL Server setup.

Once you have setup the database, you may have to Create the Jazz Database Tables.  In later releases of RTC this is done as part of the setup wizard.

Please update any site specific settings or important notes in the Installation Configuration Documentation Worksheet.
Setup Application ServerIf you are using Tomcat, read Starting the Apache Tomcat Server.  This step consists mainly in just starting the Tomcat server with the startup.bat or startup.sh script in the main Jazz server directory.

if you are using WebSphere, you should read and follow these instructions on Setting up a WebSphere Application Server.
Upload License Activation Kit: Server and Client LicensesNow upload your license activation kit, following these instructions on Managing License Keys.
Verify that the RTC Server is operating properlyValidate that you have correctly configured and set up your Jazz installation, and set up the initial environment by Running the Setup Wizard.
Post Installation StepsFollow these post-installation steps, found in Completing the Installation.

Set Up LDAP AuthenticationIf you are using LDAP for authentication, coordinate the setup of LDAP authentication with your LDAP Administrator.  For guidance on configuring your LDAP
Note: It is considered a best practice to have ALL Jazz users populated from the same LDAP field, so user IDs are consistent.  Failure to do so can lead to a single user having multiple user IDs, and can cause much greater administrative and security overhead.
Configure e-mail settingsIf you want email notifications enabled, then follow these directions for Configuring E-Mail Settings.
User Account Setup –  ImportingImport Users from an External Registry Service, and assign access permission to RTC Server Repository.

Pre-requisite: This task can only be performed by a user that belongs to the RTC_Admins repository group.
Final Option Steps

Process Configuration

Once the software has been installed and configured, you will need to perform the following steps to enable the Agile process to be used on the RTC server.
  1. Bring up the project that best represents the current best practices for software development in your organization. Open the project.
  2. Hover over the project name, and select “Create Process Template…”
  3. Create a new Process Template with the following values:
    • Name -> Should be the next value in line with the current numbering scheme. For example, if the last template was “Acme_process_v3_0”, the next template update should be “Acme_process_v3_1”. Major revisions are reserved for changes to major portions of the process functionality.
    • Process ID -> Should correspond with the name of the process template. Using the previous example, the old template was “Acme_process_v3_0”, with a Process ID of “acme.process.com.3.0”. The next template update should be “Acme_process_v3_1”, with a Process ID of “acme.process.com.3.1”.
  4. Now export the process template to a zip file. Name the zip file appropriately.
  5. Move the zip file to the new system.
  6. From an RTC client, in the RTC Administration view, import the new process template.

Project Configuration

You now need to create a new project, and tailor the process being used for use by teams.

Create a New Project

  1. Select File -> New Project Area.
  2. Enter a top level name for the new project area, as well as a description. Select next.
  3. Select the process template that you just imported. Make sure that the initialize project area radio button is selected. Then select Finish.
The new project area is now created, and you are ready to make the changes needed to ready the project for use.

Configuring the Project

Configuring the project for use by teams is a process that should take roughly 15 minutes. It involves a few different steps, and the setup used by existing projects should be used as a guide for any new projects. Where there are differences from this document, it is suggested that you update the document.

Create an Appropriate Project Timeline

  • Create the needed timelines. If you should need to modify the default timeline, then change the name and identifier for the timeline to be consistent with the project. You should then set the dates for the beginning and end of the primary iterations and release milestones.
  • Create/Modify the backlog iteration.  Again, if you need to change the backlog identification, you should change the name and identifier for the backlog iteration to be consistent with the associated project and timeline. You should then set the dates for the beginning and end of the development activity.

Create Categories for Your Teams

Each project within will rely on a default organization of teams and sub-teams within the development organization. When you create your project, a high level team was created automatically for you, and the team area will match the name of your project. You should now create categories for the different areas that the project has.  Typical categories may be UI, Database, Security, etc.  Once you have created the categories, you can then go and open the Project Area, and go to the Work Item Categories tab.  On this tab you can associate these various categories with the project teams and sub-teams that you would like to have address these.  When a work item is created, it is assigned to a category, and by making the associations on this tab, you are setting up the default assignment of work items for triage and initial evaluation.

Maintenance of Jazz Repositories

These procedures are followed on a monthly basis following the initial setup of a Jazz/Team Concert environment. These are suggested activities that can be done to help keep your environment working efficiently.

Monitoring Tasks

These tasks should be done on a relatively constant basis, as a means of identifying trouble areas in the solution architecture, enabling an administrator to take proactive action so performance, availability, and quality issues can be avoided.

Setup Administrative Dashboards

An effective Jazz administrator will often create some administrative dashboards to help in the performance of their duties.  On a typical dashboard there will be a series of widgets:
  • Server Status – This is a General > Server Status widget, which does not need to be configured any further.
  • Useful Links – This is a General > Bookmarks widget, which needs to be configured by changing the title in the appearance tab to “Useful Links”. You should then add links to the various areas where you check server health, as well as forums where you get help and support.
  • Filed Against Areas – This is a Work Items > Work Item Statistics widget, that will need to be configured. It should be based on a query that you need to create that checks all of the various areas that work items have been filed against. This query should just return a list of all project work items, for all projects, sorted on the filed against field. The query needs to be created prior to creating the widget. The widget should then be edited to do a table presentation, with the Filed Against query that you created, with the parameter set to filed against.
  • Active in Last 3 Weeks – This is a Work Items > Work Item Statistics widget, that will need to be configured. It should be based on a query that you need to create that checks all of the various areas that work items have been filed against. This query should just return a list of all project work items, for all projects, sorted on the filed against field. The query needs to be created prior to creating the widget. The widget should then be edited to do a table presentation, with the Filed Against query that you created, with the parameter set to filed against.

Review of Repository Projects

The two work item query widgets need to be observed. You are looking for projects and team areas where there has been no activity. These projects that show up in the table list, but not in the list showing activity over the last 3 weeks, are potential archival candidates. The projects that have had no activity for three weeks should get an email asking them if you can archive their project. The archival of old project data helps improve Jazz performance.

In addition, other queries for project activity may also be created and tracked. The objective is to minimize the amount of old and relatively unused data that is being indexed by the Jazz server. The archived data can still be referenced in reports based on Jazz data warehouse data.

Review of repository projects and archival candidates should be done once every 1 -2 weeks.

Monitoring of Repository Size

The size of the Jazz repository should be monitored for increases in size. This should be monitored by the DBA in charge of the backed database being used by the Jazz platform. As database tables grow, they will need to have additional space allocated, or errors and significant performance issues can occur.

In addition, the Jazz administrator should also monitor Jazz repository sizes over time, and track the relative growth of the repositories. Having this information over time, and being able to provide some predictive growth is a critical function of the Jazz administrator’s job. Using projections of the sizes of the Jazz repositories can help administrators determine when repositories may need to be split, and can help in predicting the need for additional hardware resources during annual budget cycles.

The sizes of the repository areas can be obtained by running the Latest Repository Metrics report on the Jazz server in question. The Jazz admin should take the sizing data and store it, so spreadsheets and presentations to justify additional hardware procurement can be made.

A graph of the repository area growth over time can be obtained by running the Repository Metrics report, and specifying the timeframe that you would like to see. The areas to concentrate on are the following:
  • com.ibm.team.build
  • com.ibm.team.filesystem
  • com.ibm.team.reports
  • com.ibm.team.repository
  • com.ibm.team.scm
  • com.ibm.team.workitem
  • com.ibm.team.workitem.query
The resulting graph can be confusing, which is why we often suggest collecting the overall data and keeping it in a spreadsheet format.  Review of repository size should be done once every 1 -2 weeks.

Monitoring of Server Performance

Jazz administrators should monitor server performance periodically, to ensure the consistent and efficient operating of the Jazz infrastructure. This is done from the web based client, in the administrative panels. Navigate to the server tab, and select the Status > Statistics view. You will see a series of measures displayed.

Record the names and average execution times of the web services with the five longest average times, as well as the asynchronous tasks with the five longest average times. You will need to track these over time, as they are a key indicator of environment performance for end users. Trends with constantly increasing average times indicate a degradation of performance that may need to be further monitored and corrected.

Monitoring of server performance should be done once every week.

Administrative Tasks

These are the administrative tasks that should be performed on a regular basis.

Scheduled Backups of the Repository

Jazz administrators should make sure that DBAs for the backend databases that hold the Jazz repositories are backed up on a regular basis. The frequency of the backups will determine the maximum amount of data that can be potentially lost in the case of database failure or corruption.

Every 3 months, Jazz administrators should work with DBAs to validate and time the procedures needed to restore a Jazz environment. These restorations should be done using real backups of the databases, and should be done on a standalone server, which will not interfere with production resources. The time needed to perform these recoveries will be critical in determining SLAs and in setting appropriate expectations during emergency recovery operations.

Setup, Monitoring, and Management of Licenses

License management can be done from the administrative panels utilizing the web client on the Jazz server being administered. License usage counts should be monitored to determine the number of licenses being used, for each of the various license types. License usage should then be recorded and tracked over time, so trends in usage can be discovered.

Using projections of the license usage of the Jazz repositories can help administrators determine when repositories may need to be split, and can help in predicting the need for additional Jazz based licenses during annual budget cycles.

These procedures are followed on a weekly basis following the initial setup of a Jazz/Team Concert environment.

Jazz Server Tuning


Some administrators may want to take an active role in tuning the Jazz infrastructure for best performance.  For the latest guidance on the proper tuning of a Jazz infrastructure, refer to Tuning the Rational Team Concert 2.0 Server.

Upgrades of Jazz Based Resources

These procedures are set up initially, and then performed on an as-needed basis following the initial setup of a Jazz/Team Concert environment.

Client Install Areas

Client install areas for end users need to be provisioned and kept up to date. A simple webpage with links to client downloads is sufficient. Make downloads for each of the various client types in use available, and supply both the packaged (IBM Installer) version and the zip file versions of each client.

Jazz Updates

Deployments of Jazz upgrades are simple to do, but must be planned in advance. Deployment of upgrades of the tools will require the installation of new server application software, as well as new client application software. In addition, if the upgrade of the tool results in an upgrade of the underlying Jazz Foundation, then the schema of the underlying database will change. This will require a full export and import of the repository, and this will be the most time consuming portion of the upgrade process.

Planning

Planning is important, as you need to make sure that you understand the entire upgrade process. Be aware of the best practices, and plan your upgrade accordingly. Here is a list of some very specific things to do, and things to avoid.

Things to Consider

  • Do this in steps, and reduce downtime. Your biggest concern is fitting into development schedules.
  • The chunk of time is in converting data in a repository from one schema to the next. Exports of a 22 GB DB can take 3-4 hours, and then the subsequent import can take 12-16 hours.
  • Make sure that you have PLENTY of disk space.
  • ALWAYS do database backups. Do not even attempt an upgrade without a full database backup.
  • Do migrations in a test environment first. Check the timing of the entire process, and properly set the expectations of your end users.
  • Turn off transaction logging on the DB server before you do the imports, otherwise you will go slower and rapidly fill the disk space.

Upgrade the Server Software

The first step of an upgrade involves the upgrade of the server software. You will need to shut down the currently running Jazz server, and make sure that all of the Jazz processes are shut down. Once this has been done, the server software can be upgraded.

Upgrade the Client Install Areas

Next, the new client software should be made available in the client install areas. Please make the raw zip install available as well as the eclipse package (IBM Installer) available for your users.

Upgrade the Repository

The upgrade of the repository can be done by doing the following steps:
  • Do a DB backup.
  • If not using LDAP for user authentication, save a copy of the servertomcatconftomcat-users.xml file (which contains your user authentication data).
  • Kick off the repository export (repotools export) from existing code. This produces tar file of the contents of the repository for later import.
  • Go into new code/client, and do a repository import (repotools import) into the same database. Existing tables will be dropped, any new tables will be created, and the data from the tar file will be imported into the repository.
  • If not using LDAP for user authentication, restore the saved copy of the servertomcatconftomcat-users.xml file.
When everything is done, make sure that you restart the Jazz server processes, and then let the users back in.

Backup and Restoration of Jazz Resources

The backup and restoration of RTC repositories is a simple process, but one that should be well tested. These procedures are followed on a daily basis following the initial setup of a RTC/Team Concert environment.

The process to backup the repositories requires the shutdown of the RTC server, in order to ensure that database integrity is maintained. Full backups, as opposed to incremental backups, should be performed, in order to ensure database integrity. Once the backup is completed, the RTC server processes can be restarted.

Use of NAS technologies, or high availability database technologies, can help minimize the downtime associated with periodic backups of the RTC repositories.


Disaster Recovery Considerations

The following subsections cover the major considerations that a Jazz Administrator will need to account for when creating a DR architecture and procedures.

Minimize Unplanned Outages: Use SAN or NAS Technology

The Jazz solution configuration should use SAN or NAS technologies if possible, since these technologies provide a degree of fault tolerance for disk storage resources. The Jazz repositories should reside on these resources. If virtualization is being utilized for the Jazz servers, then the virtual machine (VM) images should also be stored on this media.

In addition, NAS solutions offer utilities for doing “offline backups”, by breaking the disk mirror and then backing up the “offline” copy of the disk storage. These systems will also provide the capability to store these backups for quick restoration, as well as utilities to manipulate these backups. Well prepared Jazz installations will also make sure that these backups are moved offsite at some interval; to mitigate the risks of data loss should the data from an entire site be destroyed.

Recommendation: The Enterprise Jazz Infrastructure is expected to support the software development needs of an organization, and the consequences of losing these software assets are quite severe. Use of SAN and NAS technologies can help mitigate this risk, and will allow for backup of the Jazz based artifacts with a minimum amount of down time. While NAS may impact performance, when compared to a SAN implementation, it provides hot swapping and is more reliable. For details on the trade offs between the two technologies, see Rational Team Concert (RTC) 2.0 Sizing Guide.

Use Best Available Database Backup Technologies

Depending on the database vendor chosen, various different database backup utilities are available. If using IBM’s DB2, on-line backups are available and provide a way to keep quality backups with minimal interruption. See the DB2 documentation for details on enabling on-line backups. Some specific settings are required in the area of log file management. Even with a solid backup story, scheduling occasional downtime for a full backup is wise. On Jazz.net, we take on-line database backups nightly and in conjunction with transaction logs, we have capability to restore up to the last committed transaction. Off-line backups occur only when we do migrations or host system configuration changes.

When using an Oracle database, we recommend configuring for ARCHIVE LOG mode. One can use Oracle’s RMAN utility or Enterprise manager to take backups and schedule them on a periodic basis in conjunction with archive logs. The same tools can provide functionality to restore the database in case of failure. There might be other components that need to be backed up, like instance configuration settings, to exactly restore the database to original configuration.

In addition to backing up your Jazz repository databases, you should back up data that defines your application server configuration(s). The key items to backup from the Jazz application servers include: the jazz.war file, the workitemindex directory, the teamserver properties, the profile.ini file, the update-site directory, and the conf directory.

You also should backup the following files from your web server: server.xml, web.xml, tomcat-users.xml (Tomcat), or the WebSphere profiles when using WebSphere. WebSphere provides a backup utility for doing this.

Recommendation: Deploy the Jazz repositories on an Enterprise ready database technology. Use the functionality and features available with that technology to best minimize downtime for backup and restore operations. Use standard enterprise backup procedures and policies with respect to backup media and offsite storage. Test these technologies and procedures, and document the specific approaches and procedures in place in your environment. For details on database backup technologies and concerns, see the Deployment Guide for Jazz Team Server and the Getting Started with Rational Team Concert: A Deployment Guide.

Utilizing Redundancy and Virtualization

Some High Availability (HA) configurations that allow automatic server failover are not currently fully supported for Jazz based solutions. For more information on what HA configurations are supported, and how to configure these solutions, see Setting Up a Basic High Availability Configuration.  If you are not able or willing to deploy a suported HA configuration, then we recommend the use of redundancy in server resources, in separate physical locations, to help mitigate the risk associated with loss of service due to the loss of communications with a Jazz server.

Plan A: Designated Jazz DR Server(s)

A set of Jazz Application and Jazz Web DR servers can provide a warm backup for a server that has lost connectivity. The Jazz Application or Jazz Web server can be configured and ready to take over operations with a minimum of change. The server can be preconfigured with the proper installed software, with only the files called out in the previous section needing to be restored before the DR server can be brought online.

The server indicated as the Jazz Repository DR server will need to have a different set of backup and restore criteria, since this server is the database server that stores the data. Due to the amount of data that needs to be restored, restoration of a Jazz Repository server will take much longer to restore.

Recommendation: Using Jazz DR servers requires some additional hardware be kept ready at all times, and be underutilized. Some organizations will be hesitant to spend money on underutilized resources, but this should be looked at as a way to mitigate the risks associated with the loss of software development artifacts, and the loss of time while systems are not operational.

Plan B: Designated Virtual Machines

Using Jazz Application and Jazz Web servers resident in virtual machines provides an organization with the capability to quickly bring additional capability online. The use of virtual hardware to host these resources allows an organization to quickly and accurately bring up exact replicas of resources lost to disaster situations.

Using Jazz Repository servers hosted on virtual machines allows for some of the same benefits, however the rapid pace of software development data changing within the repositories means that some sort of restoration of the databases will need to be done. Loss of performance must be measured against the DR considerations when deciding on a Jazz solution architecture.

Recommendation: The use of virtual hardware for hosting the Jazz Application and Jazz Web components can be beneficial to the disaster recovery strategy of an organization, without a large impact to performance. Providing for the recovery from the loss of a Jazz Repository server can be done with a mix of database replication technologies as well as virtualization technologies. The relative risks, consequences, and performance impacts of an organizations decision need to be carefully weighed.

Disaster Recovery Scenarios

The following are a list of common disaster recovery scenarios. They are included here as a way to help identify the common procedures and processes needed for a proper disaster recovery plan.

Disk is lost

Disk failure can cause loss of data, and users will notice that the Jazz applications and Jazz server seem either unresponsive, or unable to store changes (when information returned to user is from the server cache). Users that are developing in Eclipse workspaces will be able to continue their work on software development assets, but no data will be stored to the Jazz repositories.

Risk Assessment: The use of SAN and NAS technologies can help reduce the risk of a single disk failure causing an unplanned outage. Expect the risk to be on par of losing a power supply.

Network is lost

The LAN is offline. Users will notice that all applications (including Jazz applications) are unresponsive. Users that are developing in Eclipse workspaces will be able to continue their work on software development assets, but no data will be stored to the Jazz repositories.

Risk Assessment: Often networks may experience significant slowdown, but they are rarely offline. If there is a network outage, it impacts all users and will be handled by IT support as a top priority problem. No special procedures by the Jazz Administrators are warranted.

Server is lost

The Jazz Application, Jazz Web, or Jazz Repository (database) server is lost . DR procedures will focus on getting users back to work using the existing infrastructure. Short-term solutions will optimize accessibility over performance.

Risk Assessment: A lost server is the probably the largest risk, since this is the event that is most likely to happen. Loss of the Jazz Web or Jazz Application servers will be able to be responded to more quickly. Loss of the Jazz Repository server will take longer, since the databases will need to be restored.

Site is lost /Silo is lost

In this scenario a site is lost or inaccessible. Recovery procedures will follow those of the server lost scenario above, but require the databases to be restored from offsite backup. A lost site will likely entail loss of ALL Jazz Server resources, and some data loss..

Risk Assessment: In case of a lost site (think fire, flood, tornado), re-establishing the development environment is not a Day 1 priority.

Disaster Recovery Process

The following are the phases of any disaster recovery process. This section outlines the high level steps needed for proper disaster recovery. A separate step-by-step disaster recovery document should be made available so administrators unfamiliar with Jazz can also perform these operations. The last subsection highlights the need to exercise these processes on a regular basis to ensure that disaster recovery processes and procedures provide adequate mitigation of the risks associated with loss of the Jazz systems and their data.

Some good examples of disaster recovery capabilities and processes can be seen in the Jazz Foundation Server Backup Details and the RRC Backup and Restore documents on the Jazz wiki site.

Notification/Activation Phase

During this phase the Jazz Administrator becomes aware of a loss of service. In some scenarios, this is detected electronically, and automated processes are kicked off.. In other scenarios the notification is more manual in nature.

Once the Jazz Administrator is aware of a potential loss of service, other impacted parties need to be notified immediately. At this point the Jazz Administrator and a other stakeholders will need to assess and identify the problem, and the correct DR procedures need to be identified and executed.

Notification Procedures

<TBD>

Damage Assessment

<TBD>

Plan Activation

<TBD>

Recovery Phase

During this phase recovery assets are put into place, and the disaster recovery procedures are completed. Recovery refers to the recovery of service for the end users and stakeholders of the Jazz infrastructure.

Sequence of Recovery Activities

<TBD>

Recovery Procedures

<TBD>

Reconstitution Phase

After service, possibly at a reduced performance or capacity, has been restored, the original issue needs to be addressed. Once the cause of the loss of service has been identified and addressed, plans for moving back to the original production systems must be made and executed. This resumption of normal operations should occur with as little impact as possible to the Jazz user community.

Primary Site Recovery

<TBD>

Primary Site Replacement

<TBD>
 

Perform Quarterly Recovery Drills

Several staff members should be trained and practiced in Disaster Recovery procedures. A regular Disaster Recovery drill enforces the training and verifies that the infrastructure and procedures are working and up-to-date.

The process to backup the repositories requires the shutdown of the Jazz server, in order to ensure that database integrity is maintained. Full backups, as opposed to incremental backups, should be performed, in order to ensure database integrity. Once the backup is completed, the Jazz server processes can be restarted.

Use of NAS technologies, or high availability database technologies, can help minimize the downtime associated with periodic backups of the Jazz repositories.

General Backup Process

The backup of any Jazz repository should follow these basic steps, for each individual Jazz repository.
  1. Inform users – Inform the users of the loss of system functionality to support backups.  Provide an expected time for resumption of operation for the repository.
  2. Shut Down Jazz Server – Once the time for doing a backup occurs, shut down the Jazz server that is connected to the repository.  Note: Any other Jazz servers that reference resources on the Jazz server being backed up, will lose the ability to access those resources during this period of time.
  3. Perform Database Backup – See the standard database procedures that are in use at your facility.
  4. Ready Database for Use – Perform the standard post-backup procedures in use at your facility to prepare a database for coming back online.
  5. Start Jazz Server – Bring the Jazz server and the application server back online for the user community.
  6. Inform users – Inform the users that the Jazz server is now available for use.

Local RTC Instances

The following is a list of all Jazz instances currently deployed, and the SLAs and backup frequencies in use for each.
  • Production
    • Instance #1 – support 24×7, response in 60 minutes – full backups done weekly
    • Instance #2 – support 24×7, response in 60 minutes – full backups done weekly
  • Training and Development
    • Test Instance #1 – support 12X5, response in 180 minutes – full backups done monthly

Process Configuration Guidance

This section provides some general guidance on how process configurations and process templates should be handled and managed.  When the organization decides to make changes to the basic processes deployed, those changes should be captured here.

Changing the Process Configuration

Rational Team Concert allows administrators and team leaders the ability to change the behavior and data associated with work items, as well as implementing process specific actions to help enforce consistent software development practices within an organization.  This is typically done by manipulating a series of checkboxes and dialogs associated with a Jazz project, on the Process Configuration tab.

Since this process implementation is stored as an XML representation, administrators and team leaders also have the ability to edit this XML directly.  NEVER edit the Process Configuration Source directly unless:
  1. You are an advanced RTC administrator
  2. You have tested the change extensively in another environment
  3. The process configuration has been exported and is stored under source control.
Whenever you make a change to the process, regardless of the method used, you should first test the change in a training area.  Once the process changes have been fully tested and validated, you can then make the change to the production area. You must then make the same change to any other Jazz instances (having an enterprise process instance has been submitted to Jazz.net as an enhancement).
 

Process Configuration Standards

Timelines

Refer to the “Project Areas” page in the web for the current timelines and teams using them.

<CUSTOMER> Work Items

  • Planning Items
    • Features
    • Epics
    • User Story
    • Technical Card
  • Execution Items
    • Defect
    • Task
    • Team Properties (used for the Burn rate reports)

Never make an execution item a planning item, this is done in the Process Configuration – it will cause incorrect reports and plans.

<CUSTOMER> Fields

Critical OTB Fields

  • Filed Against – this is the team that will be working on the work item, (for a team to appear in the drop down, the team must be setup as a category and associated with the team).
  • Planned For – the iteration that the work item is assigned to

Customized Fields

  • Application – the application the work item supports
  • Target Release – the release the work item is assigned to
  • Size Estimate – Often referred to as the T-Shirt size, it is a gross estimate

<CUSTOMER> Customized Reporting

  • Report Source Code – stored under configuration management in the Tools & Process Administration project.
  • Creating viewlets – Listed below is the process for updating the web server with the new micro reports:
    • Unzip the “Extensions” file into the <RTC install directory>/server.
    • cd to <RTC install directory>/server/Extensions/DashboardReports
    • run ./install.sh
    • Go to https://<RTC Server>/jazz/admin/cmd/requestReset
    • Stop the jazz server
    • Start the jazz server

Configuration Management of RTC source code

RTC process and report source code is stored in the Tools & Process Administration project.

Reference

Installation Configuration Documentation Worksheet


Configuration DetailSetting
Product and typeRTC Enterprise Edition v2.0.0.2
Client Access Licenses (count and type)RTC Contributor – x licenses
RTC Developer – y licenses
RTC Server EnvironmentServer Name – myjazzserver.acme.com
Server IP – aaa.bbb.ccc.ddd
Install Location – E:/JazzServer
Client Location – E:/Jazz_Clients/Eclipse/v2.0.0.2
Application Server EnvironmentType – WebSphere
Server Name – myjazzserver.acme.com
Server IP – aaa.bbb.ccc.ddd
Location – /jazzserver
UserID – jazzuser
Database Server EnvironmentDatabase type –
Version –
Server Name – myserver
Server IP – aaa.bbb.ccc.ddd
User Management MethodLDAP – Active Directory
LDAP Server – ldapmain
LDAP Server IP – aaa.bbb.ccc.ddd
Jazz Groups:
  • JazzAdmins – Contains user IDs in the Jazz Administrator group
  • JazzUsers – Contains user IDs in the Jazz Contributor and Jazz Developer groups
  • JazzUtils – Contains user IDs for Email, Chat, Jazz Connectors, Jazz Bridges, and Jazz Build
LDAP Sync Frequency – 86400 seconds (every 24 hours)
SMTP Settings SMTP Server –
SMTP Server Port –
SMTP Username –
SMTP Password –
Email From Address –
Email From Name –
SMTP Reply Address –
Use STARTTLS – false
Chat Server Settings 

Hosting Request Form

These questions help the Jazz Project Administrators, Jazz Operation Team, and the software development team learn about their project needs, and help us in the determination of an appropriate environment for the software development team.
  • Who approved this request?
  • What is your project name?
  • Requested start date for hosting?
  • Who is the primary point of contact for this project?
  • Who is the project owner or manager?
  • Who will be the RTC Project Administrator for this project?
  • What kind of project is this? (Development or work items only)
  • How many users do you expect?
  • If you moving from an existing RTC Server, please fill in the web address URL:
  • Estimated initial disk space requirements in GB:
  • Estimated long term space requirements in GB:
  • Is your project currently shared with another Group? If so, what are the other groups you work with?
  • Do you have additional requirements, such as plugins, and connectors? If so, please describe.
  • Comments, and other information.
  • What is your project or team wiki/website?

Sample Support Level Agreement (SLA)

Jazz Support Team Coverage

The Jazz support team will provide support coverage 24 hours a day, 7 days a week, for the following outage situations:
  • Rational Team Concert application issues and defects, and RTC database support issues (in tandem with the IT Support Team). An example of this is when the applications or major features are non responsive or hung. Major features are defined as:  (workitems, client connections, web UI, builds).
  • Restore of application configurations and guidance on switching to disaster recovery resources during catastrophic failures.
  • Support coverage 9×5 (CST)
The Jazz Support team is located in the Central time zone (U.S.).  ‘9×5’ Coverage means ‘standard business hours’ CST.  (National and site holidays are excluded.)  The Jazz Support Team supports these service situations:
  • Network Performance slowness. If Rational Team Concert itself is running slowly, we will investigate and do what we can, within the application environment.
  • Application support : Patches and upgrades (happen during outage periods, whenever possible)
  • Scaling/Capacity issues regarding: CPU, Memory, Disk space, The team will monitor performance and will initiate the procurement of additional resources when needed.
  • Configuration of Jazz LDAP authentication mechanisms.
  • Configuration of Email services.

Data Center Support Team Coverage

The Data Center Support Team will provide support coverage 24 hours a day, 7 days a week, for the following outage situations:
  • Hardware failures, Operating Systems issues, and Network outages.
  • Backup and restore operations (Catastrophic recovery)
The Data Center Support Team is located in the Central time zone (U.S.).  ‘The Data Center Support Team supports these service situations:
  • Network Performance slowness. If Rational Team Concert itself is running slowly, we will investigate and do what we can, within the network and data center environment.
  • OS support : Patches and upgrades (happen during outage periods, whenever possible)
  • Scaling/Capacity issues regarding: CPU, Memory, Disk space, Installation and configuration of additional resources, at the direction of the Jazz Support team.
  • Provide mechanisms for LDAP authentication.
  • Provide Email services.

Dependencies

  • LDAP authentication
  • Email services
  • Network services
  • Network hardware, firewalls, proxies, etc
  • Any special local rules that apply to the lab where the servers are housed. (access, support, etc)

Installation Planning Worksheet

The following worksheet can be used for the planning of a Jazz installation.

Determine needs and configuration details

<Customer Name>

Edition

Rational Team Concert Standard edition

Client Access license Count and Type



See Client Access License Types for a listing of actions allowed for each Client Access License type.

Developer – licenses

Contributor – zero

Build System – zero

ClearCase Connector – zero

ClearQuest Connector –zero

Jazz Server environment

See Supported server environments

Hostname: djdjdj

IP address: 10.111.111.111

Platform: Microsoft Windows 2003 Server with Service Pack 2

Application Server environment

See Supported server environments

Vendor – IBM WebSphere Application Server 6.1

Web Application Directory – Program FilesIBMJazzTeamServerserver (default)

http port: 9080 (default)

https port: 9443 (default)

Database Server environment

See Supported server environments

Vendor – IBM DB2 9.1.x

Port: 50000 (default)

Connection Type: JDBC x J2EE

JDBC Password: db2admin

JDBC Location: //localhost:50000/JAZZ:fullyMaterializeLobData=false (default)

JDBC User name: db2admin

J2EE Datasource: jdbc/mydatasource

User management method

See Supported server environments

Identity management option: LDAP

LDAP Registry Location: ldap://ldap.example.com:389

User Name: Username

Password: SecretPassword

Base User DN: o=[company],l=[your city],c=[your country]

User Property Names Mapping: userId=mail,name=cn,emailAddress=mail

Base Group DN: ou=memberlist,ou=yourgroups,o=example.com

Jazz to LDAP Group Mapping: JazzAdmins= YourGroupA, JazzUsers= YourGroupB, JazzDWAdmins= YourGroupC, JazzGuests= YourGroupD

Group Name Property: cn

Group Member Property: uniquemember

SMTP Settings for email notification

SMTP Server: smtp.example.com

SMTP Username: EmailUserName

SMTP Password: SecretPassword

Mail From Address: you@example.com

Mail From Name: Your Mail Name

SMTP Reply Address: from@example.com

SMTP Server Port: 25

Use STARTTLS: True or False

NOTE: Do you plan to enable the e-mail allowlist to restrict e-mail notifications? For example, *@example.com only allows e-mails to your company e-mail addresses.

Chat Server Settings

Software: Jabber Server

Server URL: rpmgr.myladhs.org

Port: 5222 or 5223

Client environment support

See Supported client environments

 

<edited 2023-04-06 by IBM to update terminology>

Copyright © 2010 IBM Corporation

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

Thu, 17 Jun 2010