Deploying Rational Team Concert 2.0 on WebSphere Application Server for high availability using idle standby
Introduction
Idle Standby is a failover strategy for basic high availability. The Rational Team Concert Idle Standby configuration provides for failover recovery to help ensure minimal impact on business operations during planned or unplanned server outages. Implementing the Idle Standby functionality requires purchase of the Enterprise edition of Rational Team Concert. The functionality also requires the WebSphere Application Server.Key Points
Before making the decision to use the Idle Standby configuration, the following key points need to be considered:- Rational Team Concert is licensed for use as a single-server configuration and may not be used in either a cloned or a clustered configuration, except if implemented in an Idle Standby configuration. In Idle Standby, the backup server can easily be made active if the primary server fails or if maintenance needs to be performed on the primary server. Clustering for load balancing or or anything other than implementing the Idle Standby configuration is currently not supported.
- Idle Standby configuration is not intended to provide complete support for failover. If the primary server fails or if it is intentionally taken offline, some users may need to re-authenticate (web), or wait for their client to refresh a view.
- The backup server is not intended to be run for extended periods in place of the primary server. Important – Rational Team Concert allows only a single server to be active at any one time to a RTC repository. The backup (or Idle) server therefore, is configured to never run Asynchronous (or background) tasks. If a switch is made to the backup server, work should be done to bring the primary server back up as quickly as possible.
Deployment topology
The following topology diagram illustrates the configuration for the RTC basic High Availability using Idle Standby. As you can see, the IBM HTTP Server is used to direct incoming traffic to one of the two WebSphere Application Servers, Primary Server A or Backup Server B. The WebSphere servers represent a primary and secondary node in the cluster. They are both members of the same cluster cell. In addition to the WebSphere nodes, there is an LDAP server, a file server (for Lucene index), and a Database server.
Requirements
| Server | Software | Operating System |
| IBM HTTP Server | IBM® HTTP Server 6.1.0.17+ Web server plug-ins for WAS v6.1.0.17+ !WebSphere maintenance package SDKPK85942 IBM Key Management v7.0.3.28 | Windows®, Linux® |
| WebSphere Application Server Primary Server A | WebSphere Application Server v6.1.0.19 Rational Team Concert v2.0 – Enterprise edition | Windows®, Linux® |
| WebSphere Application Server Backup Server B | WebSphere Application Server v6.1.0.19 Rational Team Concert v2.0 – Enterprise edition | Windows®, Linux® |
| Optional – File Server/shared disk | Lucene Index – full text index | Windows®, Linux® |
Setting up a basic high availability configuration
Installing and configuring the IBM HTTP Server and Web-server plug-ins
To install and configure the IBM HTTP Server and Web-server plug-ins, follow these steps:- Install the IBM HTTP server. For detail information, see http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/topic/com.ibm.websphere.base.doc/info/aes/ae/tins_webserver.html.
- Install the Web-server plug-ins. For detail information, see http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/topic/com.ibm.websphere.base.doc/info/aes/ae/tins_webplugins.html.
- Configure a Web server and an application server on separate machines (remote). For detail information, see http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/topic/com.ibm.websphere.base.doc/info/aes/ae/tins_webplugins_remotesa.html.
- To secure transmissions between the Web server and the client, enable SSL on the IBM HTTP Server. For detail information, see Guide to properly setting up SSL within the IBM HTTP Server.
Installing and configuring Rational Team Concert on Primary and Backup Servers
To install and configure two instances of Rational Team Concert on WebSphere Application Server, refer to this topic: http://publib.boulder.ibm.com/infocenter/rtc/v2r0m0/topic/com.ibm.team.install.doc/topics/t_s_server_installation_setup_WAS.htmlNote: Install one server at a time. Each server references the same database in their teamserver.properties, so make sure the first server is shut down and not attached to the repository before beginning second installation.
Additional basic HA configuration steps for both Primary and Backup servers:
The jazz.war application normally installs with a single application server as its target. With the introduction of the Web server, the jazz.war application must be modified to allow routing through the Web server. The following steps outline the necessary modifications:- In the WebSphere Console, click the jazz.war application link under Enterprise Applications.
- Select Manage Modules.
- Select the check box for the jazz.war application module.
- In the list of Clusters and Servers above, choose both the Web server and application server, then click Apply.
- Click OK, then Save changes.
- Restart the jazz.war application.
Reconfigure the RTC Application on the Primary Application Server to turn off security for the jazz.war application:
- Modify the web.xml from the WAR that was installed into WAS.
TIP: You may need to uncompress the WAR file into a temporary directory to get to the web.xml file. - Change all occurrences of “CONFIDENTIAL” to “NONE”. (there should be 3)
- Make sure WAS is running, open a browser and go to: https://localhost:9043/ibm/console/logon.jsp.
- Go to the Applications -> Enterprise Applications page.
- Select the jazz_war application and click the Update button.
- Select the Replace or add a single file radio button.
- In the “Specify the path beginning with the installed application archive file to the file to be replaced or added.” entry field enter “jazz.warWEB-INFweb.xml”.
- Click Browse and select the web.xml file that was modified in step 1.
- Click Next and follow through until the application is saved.
- Go back to the Applications -> Enterprise Applications page and stop and start the jazz_war application.
Reconfigure both the Primary and Backup Rational Team Concert servers to reference the same location for the full text index. To keep the index up to date and available to either the Primary or Backup server, update the com.ibm.team.fulltext.indexLocation in teamserver.properties on both the Primary and Backup servers to store the index on a shared drive. Modify the following property in the teamserver.properties file on the Primary and Backup servers:
- On Windows, an example of this property value is:
- On Linux, an example of this property setting is:
Additional basic HA configuration steps for the Backup server:
To avoid any possible data contention between the two running Rational Team Concert servers, Asynchronous (or background) tasks must be turned off on the Backup server.- Add the following line to the teamserver.properties file on the Backup server:
com.ibm.team.repository.scheduler.migration.mode.enabled=true
- Restart the jazz.war application on the Backup server.
Edit the Web Server plugin_cfg.xml file for RTC Idle Standby
Each time a WebSphere Application Server is configured to route requests through a Web server to an application server, the Web server plugin.xml is updated with that particular Application Server connection information. Your Web server should now have partially configured the plugin-cfg.xml file. Replace and then edit the following section of the plugin-cfg.xml on the Web server to complete the configuration. This plugin-cfg.xml file resides in the Web server’s pluginconfigwebserver1 folder (where webserver1 is the name you gave to your webserver in the Installing and configuring the IBM HTTP Server and Web-server plug-ins section above).<ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" IgnoreAffinityRequests="true" LoadBalance="Round Robin" Name="RTC_basicHA_Cluster" RetryInterval="60" PostBufferSize="64" PostSizeLimit="-1" RemoveSpecialHeaders="true">
<Server LoadBalanceWeight="1" ConnectTimeout="0" ExtendedHandshake="false" MaxConnections="-1" Name="PrimaryNode01_server1" ServerIOTimeout="0" WaitForContinue="false">
<Transport Hostname="primary.hostname.company.com" Port="9080" Protocol="http"/>
</Server>
<Server LoadBalanceWeight="0" ConnectTimeout="0" ExtendedHandshake="false" MaxConnections="-1" Name="BackupNode01_server1" ServerIOTimeout="0" WaitForContinue="false">
<Transport Hostname="backup.hostname.company.com" Port="9080" Protocol="http"/>
</Server>
</ServerCluster>
<UriGroup Name="default_host_RTC_basicHA_Cluster_URIs">
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/jazz/*"/>
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/snoop/*"/>
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/hello"/>
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/hitcount"/>
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="*.jsp"/>
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="*.jsv"/>
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="*.jsw"/>
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/j_security_check"/>
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/ibm_security_logout"/>
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/servlet/*"/>
<Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/ivt/*"/>
</UriGroup>
<Route ServerCluster="RTC_basicHA_Cluster" UriGroup="default_host_RTC_basicHA_Cluster_URIs" VirtualHostGroup="default_host"/>
Verify setup and failover ability
To verify the manual failover ability of the WebSphere Application Server, edit the plugin-cfg.xml file on the Web Server so that the PrimaryNode01 _server1 has a LoadBalanceWeight =”0″ and the BackupNode01_server1 has a LoadBalanceWeight =”1″. Save the plugin-cfg.xml file. Important: Because “true” clustering and load balancing is not yet supported, at no time can both the Primary and Backup servers have non-zero for LoadBalanceWeight.- With both Primary and Backup servers online, run the WebSphere sample Snoop servlet to get the name of the server that is handling the request.
- Invoke the Snoop servlet from an HTML browser using the URL: https://webserver/snoop.
- The Request Information that is displayed will show the host serving the request as the Local host – in this case, the Server with the LoadBalanceWeight =1 should be listed.
- Try trading the LoadBalanceWeight between the Primary and Backup server and note which server handles the Snoop servlet request.
Detecting a failure of the Primary Server
To truly have high availability, you need to know when your primary server is down. This is especially important for this basic HA solution which currently does not allow for the automatic failover of primary to backup. The process of detecting a down server is a critical but tricky one. There are many factors that could contribute to the perception that a server is down including, network problems, configuration problems, application overloading, or even user error. Whatever solution you choose, you must ensure that the alert is as instantaneous as possible. Solutions such as wget for Linux, or real-time monitoring tools such as IBM ®Tivoli Monitor for WebSphere ® would work well in this case. In the future, when automatic failover and clustering become available, the need for a timely alert status will become less of an issue.Manual failover process to Idle Standby Backup Server
If you are alerted to a down primary server, or if you suspect your server is experiencing problems, and needs maintenance, the first priority is to redirect clients to a backup server so that client applications can do their work with as little interruption as possible. Once that failover succeeds, you must repair whatever went wrong on the primary application server so that it can be reintegrate back into the solution. Repairing the failed application server may just mean restarting it. If you suspect the repair or maintenance of the primary application server may take some time, you should mark the primary server as unavailable during this repairing process. To mark the primary server as unavailable, you should make the following changes to the plugin-cfg.xml file on the Web server:- In the Web server plugin-cfg.xml file, locate the following line: <Server <nop>LoadBalanceWeight =”1″ <nop>ConnectTimeout =”0″ <nop>ExtendedHandshake =”false” <nop>MaxConnections =”-1″ Name=”PrimaryNode01_server1″ <nop>ServerIOTimeout =”0″ <nop>WaitForContinue =”false”><br />
- Change the <nop>LoadBalanceWeight attribute from “1” to “0”.
- Save the plugin-cfg.xml
Manual recovery process back to Primary Server
To restore the Primary Server to its operational status, the timing of the server startup and the switching of the LoadBalanceWeight attributes is important to avoid any data contention with the Primary and Backup Server talking to the repository at the same time. During the recovery process, some users may experience a brief disconnect or delay, but once the Primary Server is back up, it should resume taking requests from the Web Server. The step-by-step process to restore the Primary Server to operational status is as follows:- Primary Server is powered off, or WebSphere Application Server is not running.
- In the Web server plugin-cfg file, locate the entry for the PrimaryNode01_server1 and change the LoadBalanceWeight attribute from “0” to “1”.
- In the Web server plugin-cfg file, locate the entry for the BackupNode01_server1 and change the LoadBalanceWeight attribute from “1 to “0”.
- Save the edits to the Web server plugin-cfg.xml file.
- Shutdown the Backup Server, and when its completely down, start the Primary Server.
- When the Primary Server is back up and has resumed taking requests, you can start the Backup Server again.