Implementing application recovery: Example script for a Jazz deployment on WebSphere

Whether your Jazz applications are deployed on PowerVM using PowerHA, VMware ESXi Server using VMware HA or physical servers, below is an example of a script that can be used in a "cron Job" or "scheduled task" to detect whether Jazz application(s) is/are available or not.

In the example given below:

1. The script is run on the application server hosting the Jazz application(s)
2. The script first checks the associated properties file to see which Jazz application(s) is/are running on the server (you would specify this ahead of time)
3. The script then checks whether each of the specified Jazz applications (JTS, RTC, RQM, RRC etc.) are returning a good http response (a 200 response)
4. If any of the specified Jazz applications return anything other than a 200 response, the script then restarts the WebSphere Application Server.

The attached script is platform independent.

Files needed for this example

There are two files needed for our example to work successfully.

1. checkCLM.pl checkCLM.pl.txt (remove the .txt extension)
2. checkCLM.properties checkCLM.properties (The Properties file has been modified to check each of the Jazz applications ... JTS, CCM, RQM, RRC etc.)

checkCLM.pl is a PERL script that first reads the checkCLM.properties file to obtain information about your deployment. It then invokes curl to check the http response. If the response is anything other than 200, it invokes a server shutdown and then a server start-up.

checkCLM.properties Is a file that is called by the checkCLM.pl script. The file contains:

• The public URI
• The application context roots in your deployment (only specify the context roots of the applications running on this particular WebSphere Profile)
• The path to the WebSphere profile on your system
• The WAS service name (typically server1).

Tools necessary for the script to work

In order for the script to work, it is assumed that you have both PERL and curl installed and the directory containing these are in your PATH.

So as to avoid having to pass WAS credentials to the shutdown command, it is necessary to store these credentials as outlined in the technote: http://www-01.ibm.com/support/docview.wss?uid=swg21440555

Using the script

You can create a cron job or scheduled task that starts the checkCLM.pl script every five minutes (or whatever meets your needs) The checkCLM.pl and the checkCLM.properties files must be located in the same directory. The checkCLM.pl script will create / append a log file in the same directory called checkCLM.log.

checkCLM.properties CUSTOMIZATIONS

The attached checkCLM.properties file checks for all four of the CLM applications (JTS, CLM, RTC, and RRC) It has a public URI stem of https://clmserver:9443 The WebSphere profile hosting the applications is installed in C:\IBM\Websphere\Appserver\profiles\AppSRV01

The contents of the properties file:

PUBLIC_URI_HOSTNAME=https://clmserver:9443/
WAS_SERVICE_NAME=server1
WAS_SERVER_PROFILE_PATH=C:\IBM\Websphere\Appserver\profiles\AppSRV01
CONTEXT_ROOT_JTS=jts
CONTEXT_ROOT_RQM=qm
CONTEXT_ROOT_RM=rm
CONTEXT_ROOT_CCM=ccm

In the example above, notice the context roots of my applications (jts, qm, rm, and ccm)

If I only had JTS running on this server, the properties file would be:

PUBLIC_URI_HOSTNAME=https://clmserver:9443/
WAS_SERVICE_NAME=server1
WAS_SERVER_PROFILE_PATH=C:\IBM\Websphere\Appserver\profiles\AppSRV01
CONTEXT_ROOT_JTS=jts
CONTEXT_ROOT_RQM=
CONTEXT_ROOT_RM=
CONTEXT_ROOT_CCM=

If only one application is running on the given server, then omit only the context root for the other applications, but leave the variable.

If I only have RQM running on this server, the properties file would have:

PUBLIC_URI_HOSTNAME=https://clmserver:9443/
WAS_SERVICE_NAME=server1
WAS_SERVER_PROFILE_PATH=C:\IBM\Websphere\Appserver\profiles\AppSRV01
CONTEXT_ROOT_JTS=
CONTEXT_ROOT_RQM=qm
CONTEXT_ROOT_RM=
CONTEXT_ROOT_CCM=

It is very important for the checkCLM.pl script provied that we leave the "stems" (CONTEXT_ROOT_JTS=, CONTEXT_ROOT_CCM=, and CONTEXT_ROOT_RM= ) but we leave the context roots for them empty if they are not running on the specified server.

If I only have JTS and RTC running on this application, my properties file would have:

PUBLIC_URI_HOSTNAME=https://clmserver:9443/
WAS_SERVICE_NAME=server1
WAS_SERVER_PROFILE_PATH=C:\IBM\Websphere\Appserver\profiles\AppSRV01
CONTEXT_ROOT_JTS=jts
CONTEXT_ROOT_RQM=
CONTEXT_ROOT_RM=
CONTEXT_ROOT_CCM=ccm

If I am using context root from an older version of RQM ( a 2.x version) my properties file would have:

PUBLIC_URI_HOSTNAME=https://clmserver:9443/
WAS_SERVICE_NAME=server1
WAS_SERVER_PROFILE_PATH=C:\IBM\Websphere\Appserver\profiles\AppSRV01
CONTEXT_ROOT_JTS=
CONTEXT_ROOT_RQM=jazz
CONTEXT_ROOT_RM=
CONTEXT_ROOT_CCM=

If I were running my applications on Linux, my properties file would be something that parallels this:

PUBLIC_URI_HOSTNAME=https://clmserver:9443/
WAS_SERVICE_NAME=server1
WAS_SERVER_PROFILE_PATH=/opt/BM/Websphere/Appserver/profiles/AppSRV01
CONTEXT_ROOT_JTS=jts
CONTEXT_ROOT_RQM=qm
CONTEXT_ROOT_RM=rm
CONTEXT_ROOT_CCM=ccm

Enterprise Deployment

In an enterprise deployment, each application server would run the checkCLM.pl file and the checkCLM.properties file would be specific to the application hosted on that machine.

What can you expect when running the script?

The script creates a date stamped log file (a new log file each day).

If each of the applications is running well, your log file would look something like this:

[2014-03-14.17.26.26] INFO - Starting checkCLM script.
[2014-03-14.17.26.26] INFO - Reading properties file.
[2014-03-14.17.26.26] INFO - [Property:Value] PUBLIC_URI_HOSTNAME: https://clm2:9443/
[2014-03-14.17.26.26] INFO - [Property:Value] WAS_SERVICE_NAME: server1
[2014-03-14.17.26.26] INFO - [Property:Value] WAS_SERVER_PROFILE_PATH: C:\IBM\WebSphere\AppServer\profiles\AppSrv01
[2014-03-14.17.26.26] INFO - [Property:Value] CONTEXT_ROOT_JTS: jts
[2014-03-14.17.26.26] INFO - [Property:Value] CONTEXT_ROOT_RQM: qm
[2014-03-14.17.26.26] INFO - [Property:Value] CONTEXT_ROOT_RM: rm
[2014-03-14.17.26.26] INFO - [Property:Value] CONTEXT_ROOT_CCM: ccm
[2014-03-14.17.26.26] INFO - Closing properties file.
[2014-03-14.17.26.26] INFO - Beginning Server Response Checks:
[2014-03-14.17.26.26] INFO - Checking server response from jts/admin...
[2014-03-14.17.26.26] INFO - [JTS_RESPONSE]: https://clm2:9443/jts/admin: 200
[2014-03-14.17.26.26] INFO - Checking server response from qm/admin...
[2014-03-14.17.26.27] INFO - [QM_RESPONSE]: https://clm2:9443/qm/admin: 200
[2014-03-14.17.26.27] INFO - Checking server response from ccm/admin...
[2014-03-14.17.26.29] INFO - [CCM_RESPONSE]: https://clm2:9443/ccm/admin: 200
[2014-03-14.17.26.29] INFO - Checking server response from rm/web...
[2014-03-14.17.26.30] INFO - [RM_RESPONSE]: https://clm2:9443/rm/web: 200
[2014-03-14.17.26.30] INFO - ALL server response checks completed with response 200.
[2014-03-14.17.26.30] INFO - Cleaning up and closing log file.
[2014-03-14.17.26.30] INFO - Stopping checkCLM script.

If the script receives a 'bad' response from the http response of an application, you can expect to see something like this: (In our example, ccm is down)

-14.15.36.26] INFO - Beginning Server Response Checks:
[2014-03-14.15.36.26] INFO - Checking server response from jts/admin...
[2014-03-14.15.37.02] INFO - [JTS_RESPONSE]: https://clm2:9443/jts/admin: 200
[2014-03-14.15.37.02] INFO - Checking server response from qm/admin...
[2014-03-14.15.37.59] INFO - [QM_RESPONSE]: https://clm2:9443/qm/admin: 200
[2014-03-14.15.37.59] INFO - Checking server response from ccm/admin...
[2014-03-14.15.38.00] ERROR - [CCM_RESPONSE]: https://clm2:9443/ccm/admin: 404 (The actual log file will not have this in bold)
[2014-03-14.15.38.00] INFO - Checking server response from rm/web...
[2014-03-14.15.38.12] INFO - [RM_RESPONSE]: https://clm2:9443/rm/web: 200
[2014-03-14.15.38.12] INFO - PID File Found. Executing C:\IBM\WebSphere\AppServer\profiles\AppSrv01\bin\stopServer.bat server1.
[2014-03-14.15.39.02] INFO - [STOP SERVER RESPONSE]:
ADMU0116I: Tool information is being logged in file
C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\stopServer.log
ADMU7702I: Because server1 is registered to run as a Windows Service, the
request to stop this server will be completed by stopping the
associated Windows Service.
ADMU0116I: Tool information is being logged in file

C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\stopServer.log

ADMU0128I: Starting tool with the AppSrv01 profile

ADMU3100I: Reading configuration for server: server1

ADMU3201I: Server stop request issued. Waiting for stop status.

ADMU4000I: Server server1 stop completed.

[2014-03-14.15.39.02] INFO - WAS successfully stopped. Attempting restart...
[2014-03-14.15.39.02] INFO - Executing C:\IBM\WebSphere\AppServer\profiles\AppSrv01\bin\startServer.bat server1 [2014-03-14.15.40.52] INFO - [START SERVER RESPONSE]:
ADMU0116I: Tool information is being logged in file
C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\startServer.log
ADMU7701I: Because server1 is registered to run as a Windows Service, the
request to start this server will be completed by starting the
associated Windows Service.
ADMU0116I: Tool information is being logged in file

C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\startServer.log

ADMU0128I: Starting tool with the AppSrv01 profile

ADMU3100I: Reading configuration for server: server1

ADMU3200I: Server launched. Waiting for initialization status.

ADMU3000I: Server server1 open for e-business; process id is 1856

In the example below, I only have JTS, RRC, and RQM running on the system (no RTC / CCM)

-14.17.01.26] INFO - Beginning Server Response Checks:
[2014-03-14.17.01.26] INFO - Checking server response from jts/admin...
[2014-03-14.17.01.26] INFO - [JTS_RESPONSE]: https://clm2:9443/jts/admin: 200
[2014-03-14.17.01.26] INFO - Checking server response from qm/admin...
[2014-03-14.17.01.27] INFO - [QM_RESPONSE]: https://clm2:9443/qm/admin: 200
[2014-03-14.17.01.27] INFO - [CCM_SKIPPED]: CCM Not checked as no context root is specified in the properties file
[2014-03-14.17.01.27] INFO - Checking server response from rm/web...
[2014-03-14.17.01.28] INFO - [RM_RESPONSE]: https://clm2:9443/rm/web: 200
[2014-03-14.17.01.28] INFO - ALL server response checks completed with response 200.
[2014-03-14.17.01.28] INFO - Cleaning up and closing log file.
[2014-03-14.17.01.28] INFO - Stopping checkCLM script.
[2014-03-14.17.06.26] INFO - Starting checkCLM script.
[2014-03-14.17.06.26] INFO - Reading properties file.
[2014-03-14.17.06.26] INFO - [Property:Value] PUBLIC_URI_HOSTNAME: https://clm2:9443/
[2014-03-14.17.06.26] INFO - [Property:Value] WAS_SERVICE_NAME: server1
[2014-03-14.17.06.26] INFO - [Property:Value] WAS_SERVER_PROFILE_PATH: C:\IBM\WebSphere\AppServer\profiles\AppSrv01
[2014-03-14.17.06.26] INFO - [Property:Value] CONTEXT_ROOT_JTS: jts
[2014-03-14.17.06.26] INFO - [Property:Value] CONTEXT_ROOT_RQM: qm
[2014-03-14.17.06.26] INFO - [Property:Value] CONTEXT_ROOT_RM: rm
[2014-03-14.17.06.26] INFO - [Property:Value] CONTEXT_ROOT_CCM: NA
[2014-03-14.17.06.26] INFO - Closing properties file.
[2014-03-14.17.06.26] INFO - Beginning Server Response Checks:
[2014-03-14.17.06.26] INFO - Checking server response from jts/admin...
[2014-03-14.17.06.26] INFO - [JTS_RESPONSE]: https://clm2:9443/jts/admin: 200
[2014-03-14.17.06.26] INFO - Checking server response from qm/admin...
[2014-03-14.17.06.27] INFO - [QM_RESPONSE]: https://clm2:9443/qm/admin: 200
[2014-03-14.17.06.27] INFO - [CCM_SKIPPED]: CCM Not checked as no context root is specified in the properties file
[2014-03-14.17.06.27] INFO - Checking server response from rm/web...
[2014-03-14.17.06.28] INFO - [RM_RESPONSE]: https://clm2:9443/rm/web: 200
[2014-03-14.17.06.28] INFO - ALL server response checks completed with response 200.
[2014-03-14.17.06.28] INFO - Cleaning up and closing log file.
[2014-03-14.17.06.28] INFO - Stopping checkCLM script.

notice the log file states that CCM was not checked. That is because our properties file did not have a value of the CCM context root.

Related topics: Approaches to implementing high availability and disaster recovery for Rational Jazz environments

External links:

• None

Additional contributors: None