Change and Configuration Management clustered environment

Authors: MichaelAfshar, AlexBernstein, ChrisAustin, PrabhatGupta, YanpingChen, BreunReed
Build basis: Change and Configuration Management 6.0.4 and later

You can set up a clustered environment to host a Jazz Team Server and multiple Change and Configuration Management (CCM) nodes.

Approach to clustering

The CCM applications must be installed on multiple servers and connected by using an MQTT broker, which enables synchronization across the nodes. A load balancer is used as a front-end URL, which accepts connections and distributes the requests to one of the back-end CCM nodes. The host name of the load balancer is used as public URL for the CCM application during setup.

• The following diagram illustrates the approach to clustering:
  

To install the Change and Configuration Management application in a clustered environment, see the Interactive Installation Guide.

Modifying the MQTT advanced properties

You can set or modify the default MQTT related service properties on the application's Advanced Properties page:

1. Log in to the application (CCM) server as an administrator.
2. Click Application > Advanced Properties.
3. Search for com.ibm.team.repository.service.mqtt.internal.MqttService.
4. Click Edit in the title bar and modify each entry. Click Preview in the title bar to exit the edit mode and save your changes. The following table list all MQTT properties:

Table 1. MQTT properties

Property	Description
MQTT Broker address	The address of the MQTT (MQ Telemetry Transport) broker. If not provided, clustering support will be disabled.
Session persistence location	The location (the folder) where the "in-flight" messages will be stored if file-persistent location is enabled. Default is mqtt sub-folder under server folder.
Unique cluster name	This property can be used to uniquely name your cluster. This property must be set when there are more then one clusters communicating through the same MQTT message broker, or when there are more than one cluster-enabled application on the same cluster. If not provided, the port number of MQTT message broker will be used, but that may not be adequate.
Enable file-based persistence	Enabling file-based persistence will cause published MQTT messages to be saved, until the message is confirmed to be delivered. If a node goes down and restarted, or failed connection to MQTT broker is restored, persisted messages with QoS 1 and 2 will be resent.
Enable in-memory MQTT message	Message logging causes the last received message for each topic to be retained in memory for debug access.
MQTT message log size	Number of messages to store in the MQTT message log.
Maximum callback processing	Maximum number of concurrent background threads to use for processing incoming MQTT messages. Must be greater than minimum.
Minimum callback processing	Minimum number of concurrent background threads to use for processing incoming MQTT messages.
Queue size	This queue will hold background tasks submitted to process the incoming MQTT messages until a callback processing thread becomes available.

Configure HAProxy server as load balancer for ccm cluster/ JAS cluster/ IoT MessageSight cluster

Installation and setup instruction for HAProxy server can be found in the Interactive Installation Guide.

In the SVT Single JTS topology, HAProxy server connects to ccm cluster and JAS cluster in http mode, and connects to IoT MessageSight cluster in tcp mode. The connection information need to be configured in HAProxy_Install_Dir/haproxy.cfg file. The following example shows how to define http connection and tcp connection in haproxy.cfg file.

HAProxy_Install_Dir/haproxy.cfg

  # connect JAS cluster in http mode
  frontend jas-proxy
    bind  *:80
    bind  *:9643 ssl crt /etc/haproxy/ssl/proxy.pem no-sslv3
    log   global
    option httplog
    mode  http
    capture  cookie SERVERID len 32
    redirect scheme https if !{ ssl_fc }
    maxconn 2000 # The expected number of the users of the system.
    default_backend jas 
  backend jas
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    fullconn 1000 # if not specified, HAProxy will set this to 10% of 'maxconn' specified on the frontend
    balance leastconn
    cookie SERVERID insert indirect nocache
   server jas1 [JAS server 1 URI]:9643 minconn 100 maxconn 500 ssl check cookie jas1 verify none 
   server jas2 [JAS server 2 URI]:9643 minconn 100 maxconn 500 ssl check cookie jas2 verify none

  # connect MessageSight cluster in tcp mode
  listen MessageSight
    bind *:1883
    mode tcp
    balance leastconn
    option tcplog
    server MessageSight1 [MessageSight server 1 URI]:1883 check
    server MessageSight2 [MessageSight server 2 URI]:1883 check

After restart HAProxy server, HAProxy Stats for HAProxy Cluster1 and HAProxy Cluster2 in SVT Single JTS topology should be like the following:

Note: The MessageSight1 instance is online. Therefore, MessageSight2 is offline since it serves as non-primary standby message server in our HA configuration. On failure of the active appliance, the standby appliance activates itself by constructing the IBM MessageSight server state from the data in the store.

Configure high availability for HAProxy clusters

We use the Keepalived service and a virtual IP address that would be shared between the master and slave HAProxy nodes to setup high availability.

We use a script master_backup.sh to check status of HAProxy server nodes and to do the switch between the master and slave nodes. Example of Keepalived_Install_Dir/keepalived.conf and Keepalived_Install_Dir/master_backup.sh are as following:

Keepalived_Install_Dir/keepalived.conf

  ! Configuration File for keepalived
  vrrp_script chk_haproxy {
   script "killall -0 haproxy"
   interval 2
   weight 2
  }
  vrrp_instance VI_1 {
    state MASTER  # BACKUP on the stand-by node
    interface eno192 # To be replaced by real interface number
    virtual_router_id 11
    priority 101  # 100 on the stand-by node
    virtual_ipaddress {
       10.10.10.20/24 # To be replaced by real cluster ViP 
    }
    notify_master "/etc/keepalived/master_backup.sh MASTER"
    notify_backup "/etc/keepalived/master_backup.sh BACKUP"
    notify_fault "/etc/keepalived/master_backup.sh FAULT"
    track_script {
   chk_haproxy
    }
  }

Keepalived_Install_Dir/master_backup.sh   

  #! /bin/bash
  STATE=$1
  NOW=$(date)
  KEEPALIVED="/etc/keepalived"
  case $STATE in
   "MASTER") touch $KEEPALIVED/MASTER
        echo "$NOW Becoming MASTER" >> $KEEPALIVED/COUNTER
        /bin/systemctl start haproxy
        exit 0
        ;;
   "BACKUP") echo "$NOW Becoming BACKUP" >> $KEEPALIVED/COUNTER
        /bin/systemctl stop haproxy || killall -9 haproxy
        exit 0
        ;;
   "FAULT") echo "$NOW Becoming FAULT" >> $KEEPALIVED/COUNTER
        /bin/systemctl stop haproxy || killall -9 haproxy
        exit 0
        ;;        
   *)      echo "unknow state" >> $KEEPALIVED/COUNTER
        echo "NOW Becoming UNKNOWN" >> $KEEPALIVED/COUNTER
        exit 1
        ;;  
  esac

Configure high availability for IoT MessageSight cluster

Before configure high availability for IoT MessageSight servers, we need to add two additional vNICs to each IoT MessageSight node. One for discovery interface, and the other one for replication interface. The discovery IP should be in the same subnet as the host’s IP, and the replication IP should be in a different subnet.

For example, we can have additional vNICs like this:

Table 2. vNICs

Address	Hostname
10.10.40.1	MessageSight1
10.10.40.20	MessageSight1-discover
10.10.90.48	MessageSight1-replicate
10.10.40.2	MessageSight2
10.10.40.30	MessageSight2-discover
10.10.90.49	MessageSight2-replicate

The steps to setup IoT MessageSight cluster are as following:

1. For each IoT MessageSight node, install the non-Dockerized version of IoT MessageSight and configure it according to the Interactive Installation Guide using IoT MessageSight Web UI.
2. Disable all existing endpoints.
3. In IoT MessageSight Web UI, do the following on MessageSight1 node (the primary node):
   1. Select Server > High Availability, then click Edit in Configuration section.
   2. Configure the server as follows, then Save.

1. 1. Switch to MessageSight2(the standby node).
   2. Select Server > High Availability, then click Edit in Configuration section.
   3. Configure the server as follows, then Save.

1. 1. Switch to MessageSight1, restart the server in Clean store mode: select Server > Server Control, then click Clean store in the IoT MessageSight Server section
   2. Switch back to MessageSight2 and restart the server in Clean store mode. Make sure there is no error in node synchronization.
   3. Switch to MessageSight1, enable endpoints then restart the server in Clean store mode again.

To get more information about implementing high availability for IoT MessageSight, see the following topics in the IBM IoT MessageSight documentation:

* High Availability overview
* Configuring your system for high availability
* Configuring MessageSight by using the Web UI
* Configuring MessageSight by using REST Administration APIs

To setup communication between ccm cluster and IoT MessageSight cluster, add the following lines to CCM_Install_Dir/server/server.startup file:

CCM_Install_Dir/server/server.startup  

  JAVA_OPTS="$JAVA_OPTS -Dcom.ibm.team.repository.cluster.nodeId="To be replaced by a unique ccm node id""
  JAVA_OPTS="$JAVA_OPTS -Dcom.ibm.team.repository.service.internal.db.allowConcurrentAccess=true"
  JAVA_OPTS="$JAVA_OPTS -Dretry.count=0"
  JAVA_OPTS="$JAVA_OPTS -Dretry.wait=10"
  JAVA_OPTS="$JAVA_OPTS -Dactivation.code.ClusterSupport=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

Add the following line to CCM_Install_Dir/server/conf/ccm/teamserver.properties file:

CCM_Install_Dir/server/conf/ccmwork/teamserver.properties  

  com.ibm.team.repository.mqtt.broker.address=tcp\://[HAProxy cluster 2 visual IP]\:1883

Configure IHS as Reverse Proxy

Note: This section describes about configuring IHS only as a reverse proxy and not for load balancing between clustered nodes. You can also configure IHS to load balance between clustered nodes along with reverse proxy which will be described in next section.

1. Setup IHS proxy server on a remote web server.
2. Configure CLM Plugins for each WAS based CLM applications.
3. Configure IHS for all Liberty based Apps, and for both HAProxy clusters.
   1. In case that the non WAS based applications require different ports, define the ports in HTTPServer_Install_Dir/conf/httpd.conf file and WebSphere_Plugins_Install_Dir/config/webserver1/plugin-cfg.xml. For example, to define port 9643, add the following into httpd.conf file and plugin-cfg.xml files:

      HTTPServer_Install_Dir/conf/httpd.conf
      
          Listen 0.0.0.0:9643
          <VirtualHost *:9643>
          SSLEnable
          </VirtualHost>
          KeyFile HTTPServer_Install_Dir/ihskeys.kdb
          SSLStashFile HTTPServer_Install_Dir/ihskeys.sth
          SSLDisable
      
      WebSphere_Plugins_Install_Dir/Plugins/config/webserver1/plugin-cfg.xml
      
          <VirtualHostGroup Name="default_host">
              <VirtualHost Name="*:9443"/>
              <VirtualHost Name="*:9444"/>
              <VirtualHost Name="*:9643"/>
          </VirtualHostGroup>
            

   2. In the plugin-cfg.xml file, add access information for each of the non WAS based applications/clusters. For a cluster, only one entry is required. In the SVT Single JTS topology, we need entries for JTS, HAProxy cluster1, and HAProxy cluster2. An example is as following:

      
      WebSphere_Plugins_Install_Dir/Plugins/config/webserver1/plugin-cfg.xml
      
          <ServerCluster Name="JTS" ServerIOTimeoutRetry="-1" CloneSeparatorChange="false" LoadBalance="Round Robin" GetDWLMTable="false" PostBufferSize="0" IgnoreaproxyAffinityRequests="false" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60">
            <Server Name="jts" ConnectTimeout="0" ExtendedHandshake="false" ServerIOTimeout="900" LoadBalanceWeight="1" MaxConnections="-1" WaitForContinue="false">
              <Transport Protocol="https" Port="9443" Hostname="To be replaced by JTS’s URL" >
                <Property name="keyring" value="HTTPServer_Install_Dir/ihskeys.kdb"/>
                <Property name="stashfile" value="HTTPServer_Install_Dir/ihskeys.sth"/>
              </Transport>
            </Server>
          </ServerCluster>
          <UriGroup Name="jts_URIs">
            <Uri Name="/jts/*" AffinityURLIdentifier="jsessionid" AffinityCookie="JSESSIONID"/>
            <Uri Name="/clmhelp/*" AffinityURLIdentifier="jsessionid" AffinityCookie="JSESSIONID"/>
          </UriGroup>
          <Route VirtualHostGroup="default_host" UriGroup="jts_URIs" ServerCluster="JTS"/>
      
          <ServerCluster Name="ccm_cluster_server1" ServerIOTimeoutRetry="-1" CloneSeparatorChange="false" LoadBalance="Round Robin" GetDWLMTable="false" PostBufferSize="0" IgnoreaproxyAffinityRequests="false"  PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60">
            <Server Name="ccm_cluster_server1" ConnectTimeout="0" ExtendedHandshake="false" ServerIOTimeout="900" LoadBalanceWeight="1" MaxConnections="-1" WaitForContinue="false">
              <Transport Protocol="https" Port="9443" Hostname="To be replaced by the visual IP that defined for HAProxy cluster1" >
                <Property name="keyring" value="HTTPServer_Install_Dir/ihskeys.kdb"/>
                <Property name="stashfile" value="HTTPServer_Install_Dir/ihskeys.sth"/>
              </Transport>
            </Server>
          </ServerCluster>
          <UriGroup Name="ccm_server_Cluster_URIs">
            <Uri Name="/ccmwork/*" AffinityURLIdentifier="jsessionid" AffinityCookie="JSESSIONID"/>
          </UriGroup>
          <Route VirtualHostGroup="default_host" UriGroup="ccm_server_Cluster_URIs" ServerCluster="ccm_cluster_server1"/>
      
          <ServerCluster Name="haproxy_server1_status" ServerIOTimeoutRetry="-1" CloneSeparatorChange="false" LoadBalance="Round Robin" GetDWLMTable="false" PostBufferSize="0" IgnoreaproxyAffinityRequests="false" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60">
            <Server Name="haproxy_server1" ConnectTimeout="0" ExtendedHandshake="false" ServerIOTimeout="900" LoadBalanceWeight="1" MaxConnections="-1" WaitForContinue="false">
              <Transport Protocol="https" Port="9444" Hostname="To be replaced by the visual IP that defined for HAProxy cluster1" >
                <Property name="keyring" value="HTTPServer_Install_Dir/ihskeys.kdb"/>
                <Property name="stashfile" value="HTTPServer_Install_Dir/ihskeys.sth"/>
              </Transport>
            </Server>
          </ServerCluster>
          <UriGroup Name="haproxy_server1_Cluster_URIs">
            <Uri Name="/haproxy1_stats" AffinityURLIdentifier="jsessionid" AffinityCookie="JSESSIONID"/>
          </UriGroup>
          <Route VirtualHostGroup="default_host" UriGroup="haproxy_server1_Cluster_URIs" ServerCluster="haproxy_server1_status"/>
      
          <ServerCluster Name="jas_cluster_server1" ServerIOTimeoutRetry="-1" CloneSeparatorChange="false" LoadBalance="Round Robin" GetDWLMTable="false" PostBufferSize="0" IgnoreaproxyAffinityRequests="false" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60">
            <Server Name=" jas_server1" ConnectTimeout="0" ExtendedHandshake="false" ServerIOTimeout="900" LoadBalanceWeight="1" MaxConnections="-1" WaitForContinue="false">
              <Transport Protocol="https" Port="9643" Hostname="To be replaced by the visual IP that defined for HAProxy cluster2" >
                <Property name="keyring" value="HTTPServer_Install_Dir/ihskeys.kdb"/>
                <Property name="stashfile" value="HTTPServer_Install_Dir/ihskeys.sth"/>
              </Transport>
            </Server>
          </ServerCluster>
          <UriGroup Name=" jas_cluster_server1_URIs">
            <Uri Name="/oidc/*" AffinityURLIdentifier="jsessionid" AffinityCookie="JSESSIONID"/>
            <Uri Name="/jazzop/*" AffinityURLIdentifier="jsessionid" AffinityCookie="JSESSIONID"/>
          </UriGroup>
          <Route VirtualHostGroup="default_host" UriGroup=" jas_cluster_server1_URIs" ServerCluster="jas_cluster_server1"/>
      
          <ServerCluster Name="haproxy_server2_stats" ServerIOTimeoutRetry="-1" CloneSeparatorChange="false" LoadBalance="Round Robin" GetDWLMTable="false" PostBufferSize="0" IgnoreaproxyAffinityRequests="false" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60">
            <Server Name="haproxy_server2_stats" ConnectTimeout="0" ExtendedHandshake="false" ServerIOTimeout="900" LoadBalanceWeight="1" MaxConnections="-1" WaitForContinue="false">
              <Transport Protocol="https" Port="9444" Hostname="To be replaced by the visual IP that defined for HAProxy cluster2" >
                <Property name="keyring" value="HTTPServer_Install_Dir/ihskeys.kdb"/>
                <Property name="stashfile" value="HTTPServer_Install_Dir/ihskeys.sth"/>
              </Transport>
            </Server>
          </ServerCluster>
          <UriGroup Name="haproxy_server2_stats_Cluster_URIs">
            <Uri Name="/haproxy2_stats" AffinityURLIdentifier="jsessionid" AffinityCookie="JSESSIONID"/>
          </UriGroup>
          <Route VirtualHostGroup="default_host" UriGroup="haproxy_server2_stats_Cluster_URIs" ServerCluster="haproxy_server2_stats"/>
      
            

   3. Import certificates for each of the non WAS based applications/clusters/haproxy into IHS’s keystore.
   4. Restart IHS.

Accepting all “Content” for requests through proxy

It is important to make sure that IHS is enabled to accept the content (payload) from all HTTP requests (i.e. GET, PUT, POST, etc). If this is not enabled, then you may see “HTTP 400 Bad request” errors in the proxy server as some of the HTTP GET requests are sent with a payload and it gets rejected from proxy. So make sure your plugin-cfg.xml is set with AcceptAllContent="true" to avoid such errors.

Configure IHS for load balancing

This section provides information on how we can configure and use IHS for load balancing in a clustered environment. It will illustrate the configurations considering Liberty based CLM applications. It is important to note that if IHS is planned to be used for load balancing clustered application, then HAproxy should not be used for the same. Either of the one should be present in the topology for load balancing but not both.

1. Update the Liberty server.xml configuration file on each clustered node with the unique cloneId. Update the “httpSession” tag in server.xml file in all the clustered nodes to add “cloneId” field as shown below. It’s the clone identifier of the cluster member. Within a cluster, this identifier must be unique for each node to maintain session affinity. When set, this name overwrites the default name generated by the server. The same cloneId will then be used in the Web Plugin configuration to identify affinity requests.

   <httpSession invalidateOnUnauthorizedSessionRequestException="true" cookieSecure="true" cloneId="ccm1"/>
         

2. Update the Web Plugin plugin-cfg.xml to
   1. Specify Cluster servers and the load balancing algorithm
   2. Specify the cloneId for each server as mentioned in their server.xml in liberty respectively for managing session affinity.

      <ServerCluster CloneSeparatorChange="false" LoadBalance="Round Robin" IgnoreAffinityRequests="false" Name="ccm_server_Cluster" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60" PostBufferSize="64">
         <Server CloneID="ccm1" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="500" MaxConnections="100" Name="ccm1_node" WaitForContinue="false" ServerIOTimeout="600">
            <Transport Hostname="To be replaced by hostname of the clustered node1" Port="9443" Protocol="https"> 
               <Property Name="keyring" Value="HTTPServer_Install_Dir/ihskeys.kdb"/>
               <Property Name="stashfile" Value="HTTPServer_Install_Dir/ihskeys.sth"/> 
            </Transport>
         </Server>
         <Server CloneID="ccm2" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="499" MaxConnections="100" Name="ccm2_node" WaitForContinue="false" ServerIOTimeout="600">
            <Transport Hostname="To be replaced by hostname of the clustered node2" Port="9443" Protocol="https"> 
               <Property Name="keyring" Value="HTTPServer_Install_Dir/ihskeys.kdb"/>
               <Property Name="stashfile" Value="HTTPServer_Install_Dir/ihskeys.sth"/> 
            </Transport>
         </Server>
         <PrimaryServers>
            <Server Name="ccm1_node"/>
            <Server Name="ccm2_node"/>
         </PrimaryServers>
      </ServerCluster>
      <UriGroup Name="ccm_server_Cluster_URIs">
         <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/ccm/*"/>
      </UriGroup>
      <Route ServerCluster="ccm_server_Cluster" UriGroup="ccm_server_Cluster_URIs" VirtualHostGroup="default_host"/>
            

Important parameters used in above configuration are described below.

LoadBalance: The following values can be specified for this attribute: * Round Robin * Random The Round Robin implementation has a random starting point. The first application server is picked randomly. Round Robin is then used to pick application servers from that point forward. This implementation ensures that in multiple process-based web servers, all of the processes do not start by sending the first request to the same Application Server. The Random implementation also has a random starting point. However with this implementation all subsequent servers are also randomly selected. Therefore, the same server might get selected repeatedly while other servers remain idle. The default load balancing type is Round Robin.

LoadBalanceWeight is a starting "weight". The value is dynamically changed by the plug-in during runtime. The "weight" of a server (or clone) is lowered each time a request is assigned to that clone. When all weights for all servers drop to 0 or below, the plug-in has to readjust all of the weights so that they are above 0. Using a starting value of only 2 (default), means that the weights will get to 0 very quickly and the plug-in will constantly be readjusting the weights. Therefore, It is recommended to start with a higher LoadBalanceWeight. The IBM WebSphere Application Server administrative console will allow a value up to 20 for this. However, it is certainly possible to manually edit the plugin-cfg.xml file and specify some other value for LoadBalanceWeight that is higher than 20.

Note: At runtime, the LoadBalanceWeight of each appserver in a cluster are normalized by their highest common factor. For example, 100, 90, 80 have a common factor of 10. So, these configured weights would be divided by 10 at runtime, resulting in actual starting weights of only 10, 9, 8. Setting all clones to the same starting LoadBalanceWeight (for example: 20, 20, 20) will result in an actual starting weight of only 1 for each, because of normalization. So, it is recommended to set the weight of at least one of the clones to be off by a value of 1. For example, if there are 3 clones, you might choose the starting LoadBalanceWeights to be: 20, 20, 19. After normalization the weights will be unchanged. Recommended values = all clones the same, except one clone off by one.

Affinity requests are requests that contain a session cookie (ie. JSESSIONID). Session affinity means that all requests of the same JSESSIONID will be sent to the same Application Server. For example, if the first request is sent to clone5, then the next request from that same client (affinity request) will also be sent to clone5 regardless of the LoadBalanceWeight. If using Round Robin for the LoadBalance option, by default the affinity requests do NOT lower the "weight" (IgnoreAffinityRequests="true"). This can cause an uneven distribution across the servers in environments that make use of session affinity. But, If IgnoreAffinityRequests="false" then the weight IS lowered by each affinity request, leading to a more balanced Round Robin environment. When using Random, the affinity requests are still handled correctly (sent to same cloneid as before). But new requests are routed randomly, and the LoadBalanceWeight is not used. Note: The IgnoreAffinityRequests option is only available in the web server plug-in v6.1 or higher.

ConnectTimeout means "how long should the plug-in wait when trying to open a socket to the Application Server"? If there are streams already open and available to the Application Server, the plug-in will use one of those. However, sometimes the plug-in needs to open a new stream to the Application Server. That should not take very long, so the value for ConnectTimeout should be very small. A ConnectTimeout value of 0 means never time-out. In that case, the time-out is left up to the OS TCP layer, which is NOT ideal. It is much better to specify a small positive number (like 5 seconds). Recommended value = 5

MaxConnections Specifies the maximum number of pending connections to an application server that can be flowing through a web server process at any point in time. Specify one element for each Server. It is not used to determine when to fail-over (mark the server "down"). When a request is sent from the plug-in to the WAS appserver, it is called a "PendingRequest", until the response comes back. If the application running in WebSphere Application Server is handling requests quickly, each request will only be PENDING for a very short time. So, under ideal conditions, MaxConnections is not needed and therefore the default is (-1) meaning unlimited. However, sometimes an application may start to become overwhelmed and the application may not be able to handle the requests as quickly. Consequently Pending Requests start to build up. MaxConnections can be used to put a limit on the number of PENDING requests per server. When the MaxConnections limit is reached, the plug-in will stop sending requests to that appserver, but it is not marked down. The optimal value for MaxConnections will depend on how quickly the application and appserver respond to each request. If normal responses are returned in less than one second, it may be appropriate to set a low value for MaxConnections, like 20 or so. However, if it normally takes several seconds to get a response from the application, then it would be prudent to use a higher value for MaxConnections, like 100. Please note that if the MaxConnections limit has been reached the plug-in will not send ANY more requests to that server until responses come back for the current PENDING requests, and the pendingRequests count drops back down below the MaxConnections limit.

Best Practices: with MaxConnections="-1" use LogLevel="Stats" to monitor the pendingRequests numbers in the plug-in log, under normal conditions. Then, choose a value for MaxConnections that is significantly higher than the highest number shown in the log. This method will help you to determine a MaxConnections value that is right for your specific environment.

!ServerIOTimeout means how long should the plug-in wait for a response from the application. After the socket is opened, the plug-in sends the request to the Application Server. The application processes the request and a response is sent back to the client, through the plug-in. How long should that take? What is reasonable, based on the application? There is no single correct answer here. It depends on the application. If the application is very quick to respond, then you can use a lower value for ServerIOTimeout. However, if the application requires more time to process the request (maybe to retrieve data from a database), then you should use a higher number for ServerIOTimeout. Using a value of 0 means that the plug-in will NOT time-out the request. This is often NOT ideal. A positive value means that the plug-in will NOT mark the appserver down after a ServerIOTimeout pops. So, if you want the plug-in to continue sending requests to the timed-out appserver, use a positive value. On the other hand, a negative value means that the plug-in WILL mark the appserver down after a ServerIOTimeout pops. So, if you want the plug-in to immediately mark the appserver down and fail-over to another appserver in the same cluster, use a negative value.

Best Practices: use traces to determine the amount of time it takes for your application to respond to requests under normal conditions. Be sure to include the longest running requests that take the most time to respond. Then choose a value for ServerIOTimeout that is much larger (2X or 3X or more) than the longest response time. This method will ensure that your ServerIOTimeout is high enough to allow adequate time for the application to respond normally. Make it a negative value so that if the ServerIOTimeout pops, the plug-in will immediately mark the server down, and retry the request to a different appserver.

Note: It is important to set _IgnoreAffinityRequests="false"_ along with _LoadBalance="Round Robin"_ (for Plugin version higher than 6.1). This means that system will follow Round robin algorithm to pick up clustered server for non-sticky requests and will also reduce the load balance weight of server even if it’s an sticky routing. Other values can be fine-tuned as per application load.

To get more information about configuration parameters used in IHS and liberty, see the following documents:

* Understanding IBM HTTP Server plug-in Load Balancing in a clustered environment
* WebSphere Plug-in Session Affinity and Load Balancing
* Recommended values for web server plug-in config
* Configuration elements in plugin-cfg.xml file
* Configuration elements in the server.xml file

Distributed services

Troubleshooting the cluster

Related topics: Migrate from Traditional WebSphere to WebSphere Liberty

External links:

• Interactive Installation Guide

Additional contributors: TWikiUser, TWikiUser