Configuring IBM DB2 9.7 High Availability Database Repository for Rational Team Concert

Myroslav Palenychka, IBM
Last updated: June 23, 2010
Build basis: Rational Team Concert 2.0.0.2

Abstract

This article will describe how to configure your Rational Team server with a DB2 database server by leveraging DB2’s High Availability and Disaster Recovery (HADR) feature. This configuration will allow minimizing down time of the Rational Team Concert server from software or hardware failures that may occur on the system where the database repository resides.

Introduction

Rational Team Concert uses a relational database system as a back end repository to store all the information. Most popular relational databases such as Oracle, Microsoft SQL and DB2 are supported by Rational Team Concert. It is very important to minimize the effects of failures on the repository in order to maximize your team’s business operations. Down times occur when a database is unavailable due to planned or unplanned database server outages.

High availability systems are those systems that are available to users for business operations almost all the time. Database high availability is implemented by synchronizing and replicating data between two database servers, which are called the primary and standby servers. In cases when the primary database server becomes unavailable, all requests are re-routed to the standby server to recover from a failure with a minimal server interruption. The standby server becomes the new primary server in this case.

The High Availability and Disaster Recovery (HADR) feature is a replication service for DB2 databases that provides high availability functionality. The feature comes for free with DB2 Enterprise Server and can be purchased as an add-on capability to DB2 Express and Workgroup editions. Rational Team Concert uses a DB2 JDBC client to connect to a DB2 database. The next sections will guide the reader through all the necessary steps required to configure Rational Team Concert with a high availability DB2 database.

Software Requirements

Topology

Figure 1. Topology for RTC repository configuration with High Availability and Disaster Recovery (HADR)

Deployment Consideration

• Both primary and standby DB2 servers should use the same authentication mechanism and should be installed on the same version of operating system
• The CPU architecture should be the same on both servers. It is also recommended that the size of RAM should match on both machines so that users will not see any performance difference after a database switchover.
• The DB2 Data Server version installed on the primary and back up servers should match.
• This article discusses how database high availability prevents Rational Team Concert from experiencing database failures due to software or hardware errors. Failures can also occur on the Web Application Server side where the Rational Team Concert application is hosted, preventing access to business information. For more information on this topic, please refer to "Deploying Rational Team Concert 2.0 on WebSphere Application Server for high availability using idle standby".
• Both the primary and standby DB2 data servers should be fully licensed. For more information about licensing, please refer to “Licensing distributed DB2 9.5 servers in a high availability (HA) environment”

Setting up a DB2 database for Rational Team Concert

1. Install and configure DB2 data server on the primary and standby machines. Note that if you use DB2 Express or Workstation edition, you will need a HADR add-on. DB2 Enterprise edition comes with the HADR feature.
2. Set up Rational Team Concert with IBM DB2 server on the primary server. The step-by-step instructions on how to complete the setup can be found at http://publib.boulder.ibm.com/infocenter/rtc/v2r0m0/index.jsp. Later, this database will be recreated on the standby server.

Configuring HADR for a RTC DB2 database

1. At this point you should have your Rational Team Concert configured with an IBM DB2 database without the high availability feature. Before proceeding with configuring HADR for your database, make sure your Rational Team Concert server is not running. This will prevent you from connecting to the database while you are configuring high availability and disaster recovery for your database.
2. As depicted on figure 1, the standby and primary databases communicate via TCP/IP and thus you need to open ports on the operating systems where the databases are hosted. Ports can be specified by editing /etc/services for Linux/Unix and c:windowssystem32driversetcservices for Windows operating system types respectively.

     # HADR ports assigned by user  ha_rtc        51012/tcp  ha_rtc_int  51013/tcp  

   When choosing a port number, make sure it is unique to the system and does not create conflicts.
   
3. Now we can configure HADR parameters for your standby and primary databases. Run the following commands with appropriate information on the primary data server first. It is assumed that a DB2 database named JAZZHADR is created and ready for use.

     db2 update db cfg for JAZZHADR using hadr_local_host    <primary machine IP address>  db2 update db cfg for JAZZHADR using hadr_remote_host   <standby machine IP address>  db2 update db cfg for JAZZHADR using hadr_local_svc     ha_rtc  db2 update db cfg for JAZZHADR using hadr_remote_svc    ha_rtc_int  db2 update db cfg for JAZZHADR using hadr_remote_inst   <DB2 instance name on the standby server>  db2 update db cfg for JAZZHADR using hadr_timeout       120  db2 update db cfg for JAZZHADR using hadr_syncmode      nearsync  db2 update db cfg for JAZZHADR using logretain 	  on  db2 update db cfg for JAZZHADR using LOGINDEXBUILD 	  on  db2 update alternate server for database JAZZHADR using hostname <Standby server?s IP address>  port 50001  

4. Check the HADR configuration by running the following db2 command:

     db2 get db cfg for JAZZHADR | grep HADR  

   You will get output similar to what is shown below.

      HADR database role                                      = STANDARD   HADR local host name                  (HADR_LOCAL_HOST) = svt-hahost1-vm   HADR local service name                (HADR_LOCAL_SVC) = ha_rtc_int   HADR remote host name                (HADR_REMOTE_HOST) = svt-hahost2-vm   HADR remote service name              (HADR_REMOTE_SVC) = ha_rtc   HADR instance name of remote server  (HADR_REMOTE_INST) = db2inst1   HADR timeout value                       (HADR_TIMEOUT) = 30   HADR log write synchronization mode     (HADR_SYNCMODE) = NEARSYNC   HADR peer window duration (seconds)  (HADR_PEER_WINDOW) = 300  

5. Now we will back up the JAZZHADR database on the primary machine so that we can restore it on the standby server later. Execute the following commands on the primary database server:

     mkdir tmpdir  cd tmpdir  db2 backup db JAZZHADR  

   Expected output:

     Backup successful. The timestamp for this backup image is : 20100430151224  

6. Copy the folder that contains the database backup files to the standby server. Issue the following command on the standby server from the folder in order to restore the database:

     db2 restore db JAZZHADR replace history file  

7. Now that the JAZZHADR database is restored on the standby server, all HADR information we configured on the primary server was copied as well. The HADR variables that refer to local/remote hosts and port values need to be changed. Run the following commands on the standby server and provide the appropriate parameters for the standby machine:

     db2 update db cfg for JAZZHADR using hadr_local_host     <standby machine IP address>  db2 update db cfg for JAZZHADR using hadr_remote_host    <primary machine IP address>  db2 update db cfg for JAZZHADR using hadr_local_svc      ha_rtc_int  db2 update db cfg for JAZZHADR using hadr_remote_svc     ha_rtc  db2 update alternate server for database JAZZHADR using hostname <Primary machine IP   address> port 50001  

8. Verify that all parameters are correct on both data servers in order to start HADR properly. To start the standby database, first it must be deactivated and then HADR can be started on it. Run the following commands on your standby server:

     db2 deactivate db JAZZHADR   db2 start hadr on db JAZZHADR as standby  

9. After starting the standby database, verify that its role is STANDBY. You can do so by running the following command:

   db2 get db cfg for JAZZHADR | grep role

   Expected Output:

     HADR database role                                      = STANDBY  

10. The primary database needs to be activated first, and then HADR can be started.

      db2 activate db JAZZHADR  db2 start hadr on db JAZZHADR as primary  

11. Verify that the primary database’s role is correct.

    db2 get db cfg for JAZZHADR | grep role

    Expected Output:

    HADR database role                                      = PRIMARY

Automatic Client Re-routing with HADR and RTC

When RTC is configured with a DB2 database, it uses the DB2 JDBC java driver to connect to the repository. Since your database is configured with HADR, the JDBC driver gets an alternate (standby) data server’s information when connected to the primary data server. This allows the RTC server to switch database transactions to the alternate server when the primary server is not available. This particular functionality is called Automatic Client Re-routing.

Testing Takeover

Now that HADR is configured, you can start your Rational Team Concert server. Try to log in to the RTC server as an administrator to make sure the connection to the primary database server is established correctly

Case 1: The primary database needs to go down for hardware or software maintenance

1. In order for the primary and standby databases to change roles, you need to issue the switchover command on the standby server:

   db2 get snapshot for db on JAZZHADR | grep Role

   Expected Output:

   HADR database role                                      = PRIMARY

2. At this point, RTC’s connection to the primary database will time out and it will redirect all transactions to the alternate database server

Case 2: The primary database went down due to hardware or software failure

In this case, your primary database is not available and synchronization does not happen with the standby database. RTC cannot automatically re-direct all connections to the standby database server since its state becomes "Remote catch up pending". In order for RTC to start re-directing transactions to the standby server, a "takeover by force" command should be executed on the standby server.

db2 takeover hadr on db JAZZHADR by force

After this command is executed on the standby server, the state of the JAZZHADR database will become Primary. Now RTC will be able to connect to the new primary database.

Automatic Failover with Tivoli System Automation for Multiplatforms (SA MP)

So far we have discussed how to set up high availability database reposition for Jazz-based applications by taking advantage of the HADR replication feature of DB2. The main drawback of this solution is that the database manager has to manually execute a takeover command in order to switch database roles if there is a database failure. DB2 9.7 comes with SA MP, which in combination with HADR, detects network failures and recovers from a failing database by automatically switching the role of the standby database to primary. In this case, a database manager does not have to manually issue a "takeover by force" command on the standby database. Information how to set up SA MP with HADR can be found at (http://download.boulder.ibm.com/ibmdl/pub/software/dw/data/dm-0908hadrdb2haicu/HADR_db2haicu.pdf)