Jazz Administration Guide (2.x)
The Jazz Jumpstart Team: Dan Toczala, Technical Focal Point
Last updated: December 13, 2010
Build basis: Rational Team Concert 18.104.22.168
Disclaimer: This guide is an evolving document and incremental improvements will be published on a regular basis. Feedback is welcome. Please direct your comments, questions, and suggestions to Dan Toczala (firstname.lastname@example.org) The contents of this guide reflect the current Best Practices for Jazz Administration, as collected by Rational field teams. Much of the base content of this document comes from materials and experiences harvested from the Rational Engineering Servcies Jazz Operations Team. Their contribution to this has been quite valuable.
This is an example that can be used as a template for a Jazz/RTC Administration Guide. This is not meant as a complete product, but should serve as a template for the creation of Jazz/RTC Administration Guide that should be used to help provide a consistent level of service for the end users of a Jazz/RTC deployment. Many of the sections require the input of installation and site specific information. This is designed to be tailorable, and you will probably add and remove entire sections of this document, in order to improve the value of what is here, or to meet the documentation requirements of your organization.
Note: A later version of this guide is available for the RTC 3.0 release.
Last Updated on: June 6, 2010
- Architecture Overview
- Installation and Configuration
- Maintenance of Jazz Repositories
- Upgrades of Jazz Based Resources
- Backup and Restoration of Jazz Resources
- Disaster Recovery Considerations
- Disaster Recovery Scenarios
- Disaster Recovery Process
- General Backup Process
- Local RTC Instances
- Process Configuration Guidance
This document defines the set of tasks performed by the Rational Team Concert (RTC) administrator to keep the environment up and running effectively. It provides recommended guidelines in each administrative area. The implementation details can be found on Jazz.net, the product help files, or in the various IBM hosted forums and informational pages found on developerWorks. Some of these responsibilities are periodic; others are performed as required. The activities include:
- Setting up the Jazz/RTC servers
- Maintaining and monitoring integrations
- Backing up and restoring Jazz/RTC databases
- Periodically maintaining the data repository (archiving old data)
- Setting up, monitoring, and managing license usage
- Monitoring and tuning performance
- Installing patches and update areas
There are three elements to consider in the RTC environment: setting up the web server, setting up the database server, and optionally establishing the required integrations. These procedures are followed when doing the initial setup of a RTC Server environment. This section describes the procedures developed in each area.
The RTC server is a Java-based web application that runs within any Java EE 1.4 compliant application server. Two different application servers are supported in the 2.0 release of RTC: Apache Tomcat and IBM WebSphere Apache. The semantics of services and server-side RTC APIs are independent of the choice of application server, allowing additional application servers to be supported in future releases.
|Supported Application Servers ||Deployed |
|Apache Tomcat 5.5.23|
|WAS 22.214.171.124 with Fixpack 85942 |
IBM WebSphere Application Server 126.96.36.199
The repository is back-ended by a relational database. The 2.0 release of RTC supports three relational database systems: Apache Derby, Microsoft SQL server, IBM DB2 , and Oracle Database.
|Supported Database Servers ||Deployed |
|IBM DB2 9.5|
|Oracle Database 10g release 2|
|Apache Derby 10.3 |
|Microsoft SQL Server 2008 or 2005 SP2 |
|Supported LDAP Servers for Identity Management ||Deployed |
|Apache Tomcat 5.5.23|
|IBM Tivoli Directory Server 6.0|
|Microsoft Windows Server 2003 Active Directory|
In order to add thousands of Dashboard users a number of existing LDAP groups must be mapped to the JazzGuest group. On a nightly basis, RTC will bring those users over into RTC.
Note: The RTC Team Server user identity is case sensitive. When using LDAP for user management, turn off the case-insensitive option. Work with your server administrator or consult your product documentation to ensure that the settings are case-sensitive.
For Visual Studio users, a plugin for the Visual Studio IDE is provided which will allow users to access Jazz/RTC functionality from within the Visual Studio IDE.
|Supported Client Interfaces ||Deployed |
|Eclipse Based Client |
|Web Interface |
|Microsoft Visual Studio .NET Client |
The client installation instructions can be found online, as well as a list of the required Eclipse versions, and supported web browsers. Note that Internet Explorer 6 is NOT a supported browser.
IBM Rational Ensemble.
|Common Connectors and Bridges ||Deployed |
|ClearCase Connector |
|ClearCase Bridge |
|ClearQuest Connector |
|ClearQuest Bridge |
Hardware/Software RequirementsHere you would include information on your specific hardware and software needs. If you do not need this section, then please remove it.
Here you would include information on the installation of your specific integration. If you do not need this section, then please remove it.
Configuration InformationHere you would include information on the configuration details needed to support the integration. If you do not need this section, then please remove it.
Note: This information is supplied as a sample for you to use as a basis for your own information. If you do not need this section, then please remove it.
The ClearQuest Defect records will be synchronized with the Defect work items in RTC. The tool of record is ClearQuest. Defects are input into ClearQuest and then synchronized with RTC. Once in RTC, these defects are synchronized back with the original source. Defect records input into RTC will not be synchronized. This synchronization is a bidirectional synchronization, however a non tool enforced business rule has been put in place that all Defects will originate in the “System of Record”. The “System of Record” is ClearQuest.
The CQ Connector uses a server process, the ClearQuest Gateway, to establish communication between a CQ database and the RTC server. The ClearQuest Gateway can be connected to only one CQ user database.
Hardware / Software RequirementsThe CQ Connector is a tomcat instance running on a Windows or Linux machine.
|Component ||Version or Configuration |
|RTC Server Software||188.8.131.52 |
|RTC Server Hardware||RHEL WS4 (http://rtcserver.myorg.com) |
|ClearQuest Connector Software ||184.108.40.206 |
|ClearQuest Connector Hardware ||Windows 2003 Server (http://sample.myorg.com) |
|ClearQuest Client Software ||220.127.116.11 |
Installation InformationThe download of the CQ Connector is located in a download from https://jazz.net/downloads/rational-team-concert/. All of the installation information is contained within the zip file. There are no other pieces to install for the CQ Connector.
- Download the ClearQuest Connector .zip file.
- Extract the .zip file into the JazzInstallDir directory.
- After the installation is complete, configure the ClearQuest Connector. See Deploying ClearQuest Connector.
Configuration InformationTo configure the CQ Connector with the Setup Wizard, the Jazz Team Server and ClearQuest Gateway must be running. The ClearQuest Gateway must be running on a computer running Windows. The wizard generates a properties file, cqconnector.properties, that the ClearQuest Gateway uses. Be sure to read the documentation on Configuring the ClearQuest Connector.
Configuring the SynchronizerRead and understand the documentation on Configuring the ClearQuest Connector. Some key points to keep in mind as you do this:
- Note the name and password of your ClearQuest user account in RTC (often it is cqconnector).
- Have a specific login name and password for logging into ClearQuest for synchronization operations. The user account must have SQL editor privledges.
- Conector queries need to be public queries.
Configuration of the XML synchronization rulesThis publication will guide a user through the setup of the initial synchronization rule. However, there should not be any need to create a new rule from scratch. http://publib.boulder.ibm.com/infocenter/rtc/v2r0m0/index.jsp?topic=/com.ibm.team.connector.cq.doc/topics/c_prerequisites.html
|ClearQuest Field ||Mapping ||RTC Work item Attribute (field) ||Notes |
|ID||> ||CQ_ID ||Original ClearQuest record ID |
|Attachments||<> ||Attachments |
|State ||<> ||Status ||RTC work item states are a subset of the ClearQuest lifecycle. |
|etc. ||etc. ||etc. ||etc. |
MonitoringAfter the creation of the synchronization rules is complete, the administrator and team members should monitor the status of the rule(s). This task should take no longer 30 minutes.
|Synchronization Status ||Description |
|Conflict||Conflicts can occur when two users make nearly simultaneous changes to a ClearQuest record and its corresponding work item. The changes result in a ClearQuest field and the work item property to which it is mapped containing different values. You need to resolve the conflict so that the work item property and ClearQuest field contain the same field values.|
|Cycle||Too many incoming and outgoing synchronization operations have occurred without completing successfully.|
|Incoming error||An error occurred that prevented incoming synchronization of changes from ClearQuest from completing successfully.|
|OK||An incoming-outgoing or outgoing-incoming cycle has occurred and the resulting states (with respect to the properties being synchronized) are equivalent.|
|Outgoing error||An error occurred that prevented outgoing synchronization of changes from Rational Team Concert to ClearQuest from completing successfully.|
|Pending (blocked)||Changes could not be applied due to a dependency on another object that is not yet connected.|
|Pending Incoming||Outgoing changes (from Rational Team Concert to ClearQuest) have been applied. If the next incoming state is equivalent, the status changes to OK.|
|Pending Outgoing||Incoming changes (from ClearQuest to Rational Team Concert) have been applied. If the next outgoing state is equivalent, the status changes to OK. Uninitialized This is the initial state. Synchronization has not occurred.|
RTC Installation Instructions.
|Phase ||Description |
|Plan Installation||Make decisions on the architecture and collect information that will be required during the installation and configuration of the environment. Refer to the Installation Configuration Documentation Worksheet reference.|
|Verify Database Server Pre-Requisites||Database software is installed and running on a machine to be used as the database server. In production environments, this machine should be a different one from that on which the RTC Team Server runs. The test and evaluation environments this may reside on the same server. |
|Verify Application Server Pre-Requisites||Apache Tomcat should be properly configured when used as installed with RTC. |
WebSphere Application server software is already installed. Verify authentication case-insensitive property is turned off.
NOTE: Although WebSphere and many LDAP directories allow you to log in case-insensitive, RTC Team Server stores user records with user IDs in exact case as they are imported. When you log in to the RTC Team Server, the user record is retrieved from the list of users and the case must match exactly.
Verify that Java™ 2 Security option is turned off.
NOTE: If this option is turned on in WebSphere Application Server it will cause the RTC Team Server .war to fail to start.
Verify or update the WebSphere Application Server level.
NOTE: RTC Team Server requires WebSphere Application Server Version 6.1 with the IBM Java SDK 1.5 SR5 or later Cumulative Fix applied. The IBM Java SDK 1.5 SR6 SDK update is available from http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg24017492. With WebSphere 6.1 and the GA IBM Java SDK, RTC starts to report “proxy errors” after some number of operations.
Please update any site specific settings or important notes in the Installation Configuration Documentation Worksheet.
|Obtain Installation Files||Download RTC Team Server and Rational Team from Jazz.net. |
NOTE: Download files to be installed with Installation Manager. You can also do a zip install if you choose to do so.
|Install the RTC Server ||First read through all of the step-by-step installation instructions, then you can go through them and execute them |
Please update any site specific settings or important notes in the Installation Configuration Documentation Worksheet.
|Set Up Database Server ||If you are doing this in a demonstration environment, then you can just use a Derby database, since these are easy to setup and small. |
If you are doing this for a testing or production environment, you will need to follow the specific directions for the DB2 database setup, the Oracle database setup, or the SQL Server setup.
Once you have setup the database, you may have to Create the Jazz Database Tables. In later releases of RTC this is done as part of the setup wizard.
Please update any site specific settings or important notes in the Installation Configuration Documentation Worksheet.
|Setup Application Server ||If you are using Tomcat, read Starting the Apache Tomcat Server. This step consists mainly in just starting the Tomcat server with the startup.bat or startup.sh script in the main Jazz server directory. |
if you are using WebSphere, you should read and follow these instructions on Setting up a WebSphere Application Server.
|Upload License Activation Kit: Server and Client Licenses||Now upload your license activation kit, following these instructions on Managing License Keys. |
|Verify that the RTC Server is operating properly||Validate that you have correctly configured and set up your Jazz installation, and set up the initial environment by Running the Setup Wizard. |
|Post Installation Steps ||Follow these post-installation steps, found in Completing the Installation. |
|Set Up LDAP Authentication ||If you are using LDAP for authentication, coordinate the setup of LDAP authentication with your LDAP Administrator. For guidance on configuring your LDAP |
|Configure e-mail settings||If you want email notifications enabled, then follow these directions for Configuring E-Mail Settings. |
|User Account Setup – Importing ||Import Users from an External Registry Service, and assign access permission to RTC Server Repository. |
Pre-requisite: This task can only be performed by a user that belongs to the RTC_Admins repository group.
|Final Option Steps |
- Bring up the project that best represents the current best practices for software development in your organization. Open the project.
- Hover over the project name, and select “Create Process Template…”
- Create a new Process Template with the following values:
- Name -> Should be the next value in line with the current numbering scheme. For example, if the last template was “Acme_process_v3_0”, the next template update should be “Acme_process_v3_1”. Major revisions are reserved for changes to major portions of the process functionality.
- Process ID -> Should correspond with the name of the process template. Using the previous example, the old template was “Acme_process_v3_0”, with a Process ID of “acme.process.com.3.0”. The next template update should be “Acme_process_v3_1”, with a Process ID of “acme.process.com.3.1”.
Create a New Project
- Select File -> New Project Area.
- Enter a top level name for the new project area, as well as a description. Select next.
- Select the process template that you just imported. Make sure that the initialize project area radio button is selected. Then select Finish.
Configuring the ProjectConfiguring the project for use by teams is a process that should take roughly 15 minutes. It involves a few different steps, and the setup used by existing projects should be used as a guide for any new projects. Where there are differences from this document, it is suggested that you update the document.
Create an Appropriate Project Timeline
- Create the needed timelines. If you should need to modify the default timeline, then change the name and identifier for the timeline to be consistent with the project. You should then set the dates for the beginning and end of the primary iterations and release milestones.
- Create/Modify the backlog iteration. Again, if you need to change the backlog identification, you should change the name and identifier for the backlog iteration to be consistent with the associated project and timeline. You should then set the dates for the beginning and end of the development activity.
Create Categories for Your TeamsEach project within will rely on a default organization of teams and sub-teams within the development organization. When you create your project, a high level team was created automatically for you, and the team area will match the name of your project. You should now create categories for the different areas that the project has. Typical categories may be UI, Database, Security, etc. Once you have created the categories, you can then go and open the Project Area, and go to the Work Item Categories tab. On this tab you can associate these various categories with the project teams and sub-teams that you would like to have address these. When a work item is created, it is assigned to a category, and by making the associations on this tab, you are setting up the default assignment of work items for triage and initial evaluation.
Setup Administrative DashboardsAn effective Jazz administrator will often create some administrative dashboards to help in the performance of their duties. On a typical dashboard there will be a series of widgets:
- Server Status – This is a General > Server Status widget, which does not need to be configured any further.
- Useful Links – This is a General > Bookmarks widget, which needs to be configured by changing the title in the appearance tab to “Useful Links”. You should then add links to the various areas where you check server health, as well as forums where you get help and support.
- Filed Against Areas – This is a Work Items > Work Item Statistics widget, that will need to be configured. It should be based on a query that you need to create that checks all of the various areas that work items have been filed against. This query should just return a list of all project work items, for all projects, sorted on the filed against field. The query needs to be created prior to creating the widget. The widget should then be edited to do a table presentation, with the Filed Against query that you created, with the parameter set to filed against.
- Active in Last 3 Weeks – This is a Work Items > Work Item Statistics widget, that will need to be configured. It should be based on a query that you need to create that checks all of the various areas that work items have been filed against. This query should just return a list of all project work items, for all projects, sorted on the filed against field. The query needs to be created prior to creating the widget. The widget should then be edited to do a table presentation, with the Filed Against query that you created, with the parameter set to filed against.
Review of Repository ProjectsThe two work item query widgets need to be observed. You are looking for projects and team areas where there has been no activity. These projects that show up in the table list, but not in the list showing activity over the last 3 weeks, are potential archival candidates. The projects that have had no activity for three weeks should get an email asking them if you can archive their project. The archival of old project data helps improve Jazz performance.
In addition, other queries for project activity may also be created and tracked. The objective is to minimize the amount of old and relatively unused data that is being indexed by the Jazz server. The archived data can still be referenced in reports based on Jazz data warehouse data.
Review of repository projects and archival candidates should be done once every 1 -2 weeks.
Monitoring of Repository SizeThe size of the Jazz repository should be monitored for increases in size. This should be monitored by the DBA in charge of the backed database being used by the Jazz platform. As database tables grow, they will need to have additional space allocated, or errors and significant performance issues can occur.
In addition, the Jazz administrator should also monitor Jazz repository sizes over time, and track the relative growth of the repositories. Having this information over time, and being able to provide some predictive growth is a critical function of the Jazz administrator’s job. Using projections of the sizes of the Jazz repositories can help administrators determine when repositories may need to be split, and can help in predicting the need for additional hardware resources during annual budget cycles.
The sizes of the repository areas can be obtained by running the Latest Repository Metrics report on the Jazz server in question. The Jazz admin should take the sizing data and store it, so spreadsheets and presentations to justify additional hardware procurement can be made.
A graph of the repository area growth over time can be obtained by running the Repository Metrics report, and specifying the timeframe that you would like to see. The areas to concentrate on are the following:
Monitoring of Server PerformanceJazz administrators should monitor server performance periodically, to ensure the consistent and efficient operating of the Jazz infrastructure. This is done from the web based client, in the administrative panels. Navigate to the server tab, and select the Status > Statistics view. You will see a series of measures displayed.
Record the names and average execution times of the web services with the five longest average times, as well as the asynchronous tasks with the five longest average times. You will need to track these over time, as they are a key indicator of environment performance for end users. Trends with constantly increasing average times indicate a degradation of performance that may need to be further monitored and corrected.
Monitoring of server performance should be done once every week.
Scheduled Backups of the RepositoryJazz administrators should make sure that DBAs for the backend databases that hold the Jazz repositories are backed up on a regular basis. The frequency of the backups will determine the maximum amount of data that can be potentially lost in the case of database failure or corruption.
Every 3 months, Jazz administrators should work with DBAs to validate and time the procedures needed to restore a Jazz environment. These restorations should be done using real backups of the databases, and should be done on a standalone server, which will not interfere with production resources. The time needed to perform these recoveries will be critical in determining SLAs and in setting appropriate expectations during emergency recovery operations.
Setup, Monitoring, and Management of LicensesLicense management can be done from the administrative panels utilizing the web client on the Jazz server being administered. License usage counts should be monitored to determine the number of licenses being used, for each of the various license types. License usage should then be recorded and tracked over time, so trends in usage can be discovered.
Using projections of the license usage of the Jazz repositories can help administrators determine when repositories may need to be split, and can help in predicting the need for additional Jazz based licenses during annual budget cycles.
These procedures are followed on a weekly basis following the initial setup of a Jazz/Team Concert environment.
Jazz Server Tuning
Some administrators may want to take an active role in tuning the Jazz infrastructure for best performance. For the latest guidance on the proper tuning of a Jazz infrastructure, refer to Tuning the Rational Team Concert 2.0 Server.
PlanningPlanning is important, as you need to make sure that you understand the entire upgrade process. Be aware of the best practices, and plan your upgrade accordingly. Here is a list of some very specific things to do, and things to avoid.
Things to Consider
- Do this in steps, and reduce downtime. Your biggest concern is fitting into development schedules.
- The chunk of time is in converting data in a repository from one schema to the next. Exports of a 22 GB DB can take 3-4 hours, and then the subsequent import can take 12-16 hours.
- Make sure that you have PLENTY of disk space.
- ALWAYS do database backups. Do not even attempt an upgrade without a full database backup.
- Do migrations in a test environment first. Check the timing of the entire process, and properly set the expectations of your end users.
- Turn off transaction logging on the DB server before you do the imports, otherwise you will go slower and rapidly fill the disk space.
Upgrade the Server SoftwareThe first step of an upgrade involves the upgrade of the server software. You will need to shut down the currently running Jazz server, and make sure that all of the Jazz processes are shut down. Once this has been done, the server software can be upgraded.
Upgrade the Client Install AreasNext, the new client software should be made available in the client install areas. Please make the raw zip install available as well as the eclipse package (IBM Installer) available for your users.
Upgrade the RepositoryThe upgrade of the repository can be done by doing the following steps:
- Do a DB backup.
- If not using LDAP for user authentication, save a copy of the servertomcatconftomcat-users.xml file (which contains your user authentication data).
- Kick off the repository export (repotools export) from existing code. This produces tar file of the contents of the repository for later import.
- Go into new code/client, and do a repository import (repotools import) into the same database. Existing tables will be dropped, any new tables will be created, and the data from the tar file will be imported into the repository.
- If not using LDAP for user authentication, restore the saved copy of the servertomcatconftomcat-users.xml file.
The process to backup the repositories requires the shutdown of the RTC server, in order to ensure that database integrity is maintained. Full backups, as opposed to incremental backups, should be performed, in order to ensure database integrity. Once the backup is completed, the RTC server processes can be restarted.
Use of NAS technologies, or high availability database technologies, can help minimize the downtime associated with periodic backups of the RTC repositories.
Minimize Unplanned Outages: Use SAN or NAS TechnologyThe Jazz solution configuration should use SAN or NAS technologies if possible, since these technologies provide a degree of fault tolerance for disk storage resources. The Jazz repositories should reside on these resources. If virtualization is being utilized for the Jazz servers, then the virtual machine (VM) images should also be stored on this media.
In addition, NAS solutions offer utilities for doing “offline backups”, by breaking the disk mirror and then backing up the “offline” copy of the disk storage. These systems will also provide the capability to store these backups for quick restoration, as well as utilities to manipulate these backups. Well prepared Jazz installations will also make sure that these backups are moved offsite at some interval; to mitigate the risks of data loss should the data from an entire site be destroyed.
Recommendation: The Enterprise Jazz Infrastructure is expected to support the software development needs of an organization, and the consequences of losing these software assets are quite severe. Use of SAN and NAS technologies can help mitigate this risk, and will allow for backup of the Jazz based artifacts with a minimum amount of down time. While NAS may impact performance, when compared to a SAN implementation, it provides hot swapping and is more reliable. For details on the trade offs between the two technologies, see Rational Team Concert (RTC) 2.0 Sizing Guide.
Use Best Available Database Backup TechnologiesDepending on the database vendor chosen, various different database backup utilities are available. If using IBM’s DB2, on-line backups are available and provide a way to keep quality backups with minimal interruption. See the DB2 documentation for details on enabling on-line backups. Some specific settings are required in the area of log file management. Even with a solid backup story, scheduling occasional downtime for a full backup is wise. On Jazz.net, we take on-line database backups nightly and in conjunction with transaction logs, we have capability to restore up to the last committed transaction. Off-line backups occur only when we do migrations or host system configuration changes.
When using an Oracle database, we recommend configuring for ARCHIVE LOG mode. One can use Oracle’s RMAN utility or Enterprise manager to take backups and schedule them on a periodic basis in conjunction with archive logs. The same tools can provide functionality to restore the database in case of failure. There might be other components that need to be backed up, like instance configuration settings, to exactly restore the database to original configuration.
In addition to backing up your Jazz repository databases, you should back up data that defines your application server configuration(s). The key items to backup from the Jazz application servers include: the jazz.war file, the workitemindex directory, the teamserver properties, the profile.ini file, the update-site directory, and the conf directory.
You also should backup the following files from your web server: server.xml, web.xml, tomcat-users.xml (Tomcat), or the WebSphere profiles when using WebSphere. WebSphere provides a backup utility for doing this.
Recommendation: Deploy the Jazz repositories on an Enterprise ready database technology. Use the functionality and features available with that technology to best minimize downtime for backup and restore operations. Use standard enterprise backup procedures and policies with respect to backup media and offsite storage. Test these technologies and procedures, and document the specific approaches and procedures in place in your environment. For details on database backup technologies and concerns, see the Deployment Guide for Jazz Team Server and the Getting Started with Rational Team Concert: A Deployment Guide.
Utilizing Redundancy and VirtualizationSome High Availability (HA) configurations that allow automatic server failover are not currently fully supported for Jazz based solutions. For more information on what HA configurations are supported, and how to configure these solutions, see Setting Up a Basic High Availability Configuration. If you are not able or willing to deploy a suported HA configuration, then we recommend the use of redundancy in server resources, in separate physical locations, to help mitigate the risk associated with loss of service due to the loss of communications with a Jazz server.
Plan A: Designated Jazz DR Server(s)A set of Jazz Application and Jazz Web DR servers can provide a warm backup for a server that has lost connectivity. The Jazz Application or Jazz Web server can be configured and ready to take over operations with a minimum of change. The server can be preconfigured with the proper installed software, with only the files called out in the previous section needing to be restored before the DR server can be brought online.
The server indicated as the Jazz Repository DR server will need to have a different set of backup and restore criteria, since this server is the database server that stores the data. Due to the amount of data that needs to be restored, restoration of a Jazz Repository server will take much longer to restore.
Recommendation: Using Jazz DR servers requires some additional hardware be kept ready at all times, and be underutilized. Some organizations will be hesitant to spend money on underutilized resources, but this should be looked at as a way to mitigate the risks associated with the loss of software development artifacts, and the loss of time while systems are not operational.
Plan B: Designated Virtual MachinesUsing Jazz Application and Jazz Web servers resident in virtual machines provides an organization with the capability to quickly bring additional capability online. The use of virtual hardware to host these resources allows an organization to quickly and accurately bring up exact replicas of resources lost to disaster situations.
Using Jazz Repository servers hosted on virtual machines allows for some of the same benefits, however the rapid pace of software development data changing within the repositories means that some sort of restoration of the databases will need to be done. Loss of performance must be measured against the DR considerations when deciding on a Jazz solution architecture.
Recommendation: The use of virtual hardware for hosting the Jazz Application and Jazz Web components can be beneficial to the disaster recovery strategy of an organization, without a large impact to performance. Providing for the recovery from the loss of a Jazz Repository server can be done with a mix of database replication technologies as well as virtualization technologies. The relative risks, consequences, and performance impacts of an organizations decision need to be carefully weighed.
Disk is lostDisk failure can cause loss of data, and users will notice that the Jazz applications and Jazz server seem either unresponsive, or unable to store changes (when information returned to user is from the server cache). Users that are developing in Eclipse workspaces will be able to continue their work on software development assets, but no data will be stored to the Jazz repositories.
Risk Assessment: The use of SAN and NAS technologies can help reduce the risk of a single disk failure causing an unplanned outage. Expect the risk to be on par of losing a power supply.
Network is lostThe LAN is offline. Users will notice that all applications (including Jazz applications) are unresponsive. Users that are developing in Eclipse workspaces will be able to continue their work on software development assets, but no data will be stored to the Jazz repositories.
Risk Assessment: Often networks may experience significant slowdown, but they are rarely offline. If there is a network outage, it impacts all users and will be handled by IT support as a top priority problem. No special procedures by the Jazz Administrators are warranted.
Server is lostThe Jazz Application, Jazz Web, or Jazz Repository (database) server is lost . DR procedures will focus on getting users back to work using the existing infrastructure. Short-term solutions will optimize accessibility over performance.
Risk Assessment: A lost server is the probably the largest risk, since this is the event that is most likely to happen. Loss of the Jazz Web or Jazz Application servers will be able to be responded to more quickly. Loss of the Jazz Repository server will take longer, since the databases will need to be restored.
Site is lost /Silo is lostIn this scenario a site is lost or inaccessible. Recovery procedures will follow those of the server lost scenario above, but require the databases to be restored from offsite backup. A lost site will likely entail loss of ALL Jazz Server resources, and some data loss..
Risk Assessment: In case of a lost site (think fire, flood, tornado), re-establishing the development environment is not a Day 1 priority.
Some good examples of disaster recovery capabilities and processes can be seen in the Jazz Foundation Server Backup Details and the RRC Backup and Restore documents on the Jazz wiki site.
Notification/Activation PhaseDuring this phase the Jazz Administrator becomes aware of a loss of service. In some scenarios, this is detected electronically, and automated processes are kicked off.. In other scenarios the notification is more manual in nature.
Once the Jazz Administrator is aware of a potential loss of service, other impacted parties need to be notified immediately. At this point the Jazz Administrator and a other stakeholders will need to assess and identify the problem, and the correct DR procedures need to be identified and executed.
Recovery PhaseDuring this phase recovery assets are put into place, and the disaster recovery procedures are completed. Recovery refers to the recovery of service for the end users and stakeholders of the Jazz infrastructure.
Sequence of Recovery Activities<TBD>
Reconstitution PhaseAfter service, possibly at a reduced performance or capacity, has been restored, the original issue needs to be addressed. Once the cause of the loss of service has been identified and addressed, plans for moving back to the original production systems must be made and executed. This resumption of normal operations should occur with as little impact as possible to the Jazz user community.
Primary Site Recovery<TBD>
Primary Site Replacement<TBD>
Perform Quarterly Recovery DrillsSeveral staff members should be trained and practiced in Disaster Recovery procedures. A regular Disaster Recovery drill enforces the training and verifies that the infrastructure and procedures are working and up-to-date.
The process to backup the repositories requires the shutdown of the Jazz server, in order to ensure that database integrity is maintained. Full backups, as opposed to incremental backups, should be performed, in order to ensure database integrity. Once the backup is completed, the Jazz server processes can be restarted.
Use of NAS technologies, or high availability database technologies, can help minimize the downtime associated with periodic backups of the Jazz repositories.
- Inform users – Inform the users of the loss of system functionality to support backups. Provide an expected time for resumption of operation for the repository.
- Shut Down Jazz Server – Once the time for doing a backup occurs, shut down the Jazz server that is connected to the repository. Note: Any other Jazz servers that reference resources on the Jazz server being backed up, will lose the ability to access those resources during this period of time.
- Perform Database Backup – See the standard database procedures that are in use at your facility.
- Ready Database for Use – Perform the standard post-backup procedures in use at your facility to prepare a database for coming back online.
- Start Jazz Server – Bring the Jazz server and the application server back online for the user community.
- Inform users – Inform the users that the Jazz server is now available for use.
- Instance #1 – support 24×7, response in 60 minutes – full backups done weekly
- Instance #2 – support 24×7, response in 60 minutes – full backups done weekly
- Training and Development
- Test Instance #1 – support 12X5, response in 180 minutes – full backups done monthly
Since this process implementation is stored as an XML representation, administrators and team leaders also have the ability to edit this XML directly. NEVER edit the Process Configuration Source directly unless:
- You are an advanced RTC administrator
- You have tested the change extensively in another environment
- The process configuration has been exported and is stored under source control.
TimelinesRefer to the “Project Areas” page in the web for the current timelines and teams using them.
<CUSTOMER> Work Items
- Planning Items
- User Story
- Technical Card
- Execution Items
- Team Properties (used for the Burn rate reports)
Never make an execution item a planning item, this is done in the Process Configuration – it will cause incorrect reports and plans.
Critical OTB Fields
- Filed Against – this is the team that will be working on the work item, (for a team to appear in the drop down, the team must be setup as a category and associated with the team).
- Planned For – the iteration that the work item is assigned to
- Application – the application the work item supports
- Target Release – the release the work item is assigned to
- Size Estimate – Often referred to as the T-Shirt size, it is a gross estimate
<CUSTOMER> Customized Reporting
- Report Source Code – stored under configuration management in the Tools & Process Administration project.
- Creating viewlets – Listed below is the process for updating the web server with the new micro reports:
- Unzip the “Extensions” file into the <RTC install directory>/server.
- cd to <RTC install directory>/server/Extensions/DashboardReports
- run ./install.sh
- Go to https://<RTC Server>/jazz/admin/cmd/requestReset
- Stop the jazz server
- Start the jazz server
Configuration Management of RTC source codeRTC process and report source code is stored in the Tools & Process Administration project.
|Configuration Detail ||Setting |
| Product and type || RTC Enterprise Edition v18.104.22.168 |
| Client Access Licenses (count and type) ||RTC Contributor – x licenses |
RTC Developer – y licenses
|RTC Server Environment ||Server Name – myjazzserver.acme.com |
Server IP – aaa.bbb.ccc.ddd
Install Location – E:/JazzServer
Client Location – E:/Jazz_Clients/Eclipse/v22.214.171.124
|Application Server Environment ||Type – WebSphere |
Server Name – myjazzserver.acme.com
Server IP – aaa.bbb.ccc.ddd
Location – /jazzserver
UserID – jazzuser
|Database Server Environment ||Database type – |
Server Name – myserver
Server IP – aaa.bbb.ccc.ddd
| User Management Method ||LDAP – Active Directory |
LDAP Server – ldapmain
LDAP Server IP – aaa.bbb.ccc.ddd
| SMTP Settings ||SMTP Server – |
SMTP Server Port –
SMTP Username –
SMTP Password –
Email From Address –
Email From Name –
SMTP Reply Address –
Use STARTTLS – false
| Chat Server Settings |
- Who approved this request?
- What is your project name?
- Requested start date for hosting?
- Who is the primary point of contact for this project?
- Who is the project owner or manager?
- Who will be the RTC Project Administrator for this project?
- What kind of project is this? (Development or work items only)
- How many users do you expect?
- If you moving from an existing RTC Server, please fill in the web address URL:
- Estimated initial disk space requirements in GB:
- Estimated long term space requirements in GB:
- Is your project currently shared with another Group? If so, what are the other groups you work with?
- Do you have additional requirements, such as plugins, and connectors? If so, please describe.
- Comments, and other information.
- What is your project or team wiki/website?
Jazz Support Team CoverageThe Jazz support team will provide support coverage 24 hours a day, 7 days a week, for the following outage situations:
- Rational Team Concert application issues and defects, and RTC database support issues (in tandem with the IT Support Team). An example of this is when the applications or major features are non responsive or hung. Major features are defined as: (workitems, client connections, web UI, builds).
- Restore of application configurations and guidance on switching to disaster recovery resources during catastrophic failures.
- Support coverage 9×5 (CST)
- Network Performance slowness. If Rational Team Concert itself is running slowly, we will investigate and do what we can, within the application environment.
- Application support : Patches and upgrades (happen during outage periods, whenever possible)
- Scaling/Capacity issues regarding: CPU, Memory, Disk space, The team will monitor performance and will initiate the procurement of additional resources when needed.
- Configuration of Jazz LDAP authentication mechanisms.
- Configuration of Email services.
Data Center Support Team CoverageThe Data Center Support Team will provide support coverage 24 hours a day, 7 days a week, for the following outage situations:
- Hardware failures, Operating Systems issues, and Network outages.
- Backup and restore operations (Catastrophic recovery)
- Network Performance slowness. If Rational Team Concert itself is running slowly, we will investigate and do what we can, within the network and data center environment.
- OS support : Patches and upgrades (happen during outage periods, whenever possible)
- Scaling/Capacity issues regarding: CPU, Memory, Disk space, Installation and configuration of additional resources, at the direction of the Jazz Support team.
- Provide mechanisms for LDAP authentication.
- Provide Email services.
- LDAP authentication
- Email services
- Network services
- Network hardware, firewalls, proxies, etc
- Any special local rules that apply to the lab where the servers are housed. (access, support, etc)
Determine needs and configuration details
Rational Team Concert Standard edition
Client Access license Count and Type
See Client Access License Types for a listing of actions allowed for each Client Access License type.
Developer – licenses
Contributor – zero
Build System – zero
ClearCase Connector – zero
ClearQuest Connector –zero
Jazz Server environment
IP address: 10.111.111.111
Platform: Microsoft Windows 2003 Server with Service Pack 2
Application Server environment
Vendor – IBM WebSphere Application Server 6.1
Web Application Directory – Program FilesIBMJazzTeamServerserver (default)
http port: 9080 (default)
https port: 9443 (default)
Database Server environment
Vendor – IBM DB2 9.1.x
Port: 50000 (default)
Connection Type: JDBC x J2EE
JDBC Password: db2admin
JDBC Location: //localhost:50000/JAZZ:fullyMaterializeLobData=false (default)
JDBC User name: db2admin
J2EE Datasource: jdbc/mydatasource
User management method
Identity management option: LDAP
LDAP Registry Location: ldap://ldap.example.com:389
User Name: Username
Base User DN: o=[company],l=[your city],c=[your country]
User Property Names Mapping: userId=mail,name=cn,emailAddress=mail
Base Group DN: ou=memberlist,ou=yourgroups,o=example.com
Jazz to LDAP Group Mapping: JazzAdmins= YourGroupA, JazzUsers= YourGroupB, JazzDWAdmins= YourGroupC, JazzGuests= YourGroupD
Group Name Property: cn
Group Member Property: uniquemember
SMTP Settings for email notification
SMTP Server: smtp.example.com
SMTP Username: EmailUserName
SMTP Password: SecretPassword
Mail From Address: email@example.com
Mail From Name: Your Mail Name
SMTP Reply Address: firstname.lastname@example.org
SMTP Server Port: 25
Use STARTTLS: True or False
NOTE: Do you plan to enable the e-mail whitelist to restrict e-mail notifications? For example, *@example.com only allows e-mails to your company e-mail addresses.
Chat Server Settings
Software: Jabber Server
Server URL: rpmgr.myladhs.org
Port: 5222 or 5223
Client environment support
Copyright © 2010 IBM Corporation