PerformanceClusteredJAS < Deployment

r21 - 2024-08-15 - 14:34:43 - TimFeeneyYou are here: TWiki >
Deployment Web > DeploymentPlanningAndDesign > PerformanceDatasheetsAndSizingGuidelines > PerformanceClusteredJAS
<div id="header-title" style="padding: 10px 15px; border-width:1px; border-style:solid; border-color:#FFD28C; background-image: url(<nop>https://jazz.net/wiki/pub/Deployment/WebPreferences/TLASE.jpg); background-size: cover; font-size:120%">
---+!! Setting up a cluster of Jazz Authorization Servers<img src="https://jazz.net/wiki/pub/Deployment/WebPreferences/new.png" alt="new.png" width="50" height="50" align="right"> 
%DKGRAY% Authors: Main.VaughnRokosz, Main.HongyanHuo <br>
Build basis: 6.0.2
%ENDCOLOR%</div></sticky>

<!-- Page contents top of page on right hand side in box -->
<sticky><div style="float:right; border-width:1px; border-style:solid; border-color:#DFDFDF; background-color:#F6F6F6; margin:0 0 15px 15px; padding: 0 15px 0 15px;">
%TOC{title="Page contents"}%
</div></sticky>

<sticky><div style="margin:15px;"></sticky>

This article shows you how to set up a cluster of Jazz Authorization Servers.    

The [[https://jazz.net/library/article/75][Jazz Authorization server]] allows for single sign-on across your Jazz applications.    Users can log in once, and then move between Jazz applications like Rational Team Concert or Rational Quality Manager without being prompted for credentials.    

If you have an active Jazz deployment, you can reduce the risk of outages by clustering your Jazz Authorization Server.    If you only have one JAS instance, then no one can log in if that instance is down.   But if you have a cluster, login requests can automatically fail over to other JAS instances.

---++ An overview of the approach

The diagram below illustrates the approach to clustering.    You start by setting up several Jazz Authorization Servers (JAS), and configuring each one to use a common LDAP server.   You additionally must configure each JAS to use a common database server (by default, JAS stores information locally in a Derby database, so for a cluster, you need to be able to share information between JAS instances via a common database server).    Finally, you set up a network dispatcher to distribute requests across your JAS instances.    In this article, we'll use the IBM HTTP Server (IHS) as the network dispatcher.

After you've got your JAS cluster running, you can configure your Jazz servers to use it (although there are some limitations if you have an existing non-clustered JAS already set up in your environment).   

 <img src="https://jazz.net/wiki/pub/Deployment/PerformanceClusteredJAS/Overview.png" alt="Overview.png"> 

This article focuses on clustering only JAS.    You can also set up your LDAP or database server in high availability mode.   Please refer to the documentation for your specific LDAP or database product if you are interested in doing that.

---++ Installing the Jazz Authorization Server

Your first step is to install the Jazz Authorization Server on at least two separate systems.   

You can download the Jazz Authorization Server from jazz.net (release 6.0 and later).    You install it using Installation Manager.

Configure each JAS instance to use a [[https://www.ibm.com/support/knowledgecenter/SSCP65_6.0.1/com.ibm.jazz.install.doc/topics/t_jsasso_jas_user_mgmt.html][common LDAP server by following the instructions in the knowledge center]].

After you've completed the installation, move on to the next step:    configuring persistence.

---++ Configuring persistence for the Jazz Authorization Server

When you first install the Jazz Authorization server, it comes configured to use a local Derby database for storing information.    That won't work in a clustered environment, since that information won't be available to all the instances.    In order to cluster JAS instances, you need to set up a database server which can be shared by all JAS instances.    The basic steps are:
   * Create database tables on a database server which all JAS instances can access
   * Update the JAS configuration file (appConfig.xml) to use the database server

Out of the box, the Authorization Server is bundled with Derby. We do not support customizations to the embedded derby and it is not sufficient to use for clustering your application.

The following sections will show how to do that for both Oracle and Db2, and will provide sample SQL scripts that can create the necessary tables.   But note that you will need to customize these scripts for your own environment.

You can find some general information on [[https://www.ibm.com/support/knowledgecenter/en/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/cwlp_oauth_db2.html][using Db2 for persistent OAuth services]] in the Liberty documentation.    However, some of those instructions are incorrect;   we've provided corrections here where needed.   

---+++ Note: !TokenString Size Specification. 

In all the .sql scripts (provided against Db2/Oracle/MS SQL), the size of the !TokenString provided in the Script needs individual assistance. The reason for this custom requirement, is that the size of the !TokenString, is not just dependent on how many Jazz groups that a user is a Member. In a corporate LDAP, it would be very difficult to know what is the maximum number of LDAP groups a user. 

More the user associations with different LDAP groups, bigger can be the size of the !TokenString being generated. Hence, we cannot deduce the optimal size for this column in the script. However, after experiencing this issue with clients using different Enterprise DB vendors, after consulting with Liberty development team, we have been able to get to an understanding, that this size could grow even upto 30K. 

If users have, lesser number Group associations, the default limit of Varchar(2048) might Suffice. But if this association is more, one could extend the default size by changing upto Varchar(4000) In Oracle. However, the same type in Db2 could extend upto 20K, and for MS SQL, one would need the type to be changed to NVARCHAR(MAX). 

Different databases have different upper limits for this Attribute type. Mentioned below are alternatives for each of the DB Vendors, when the default limit of 2048 becomes insufficient:

*Oracle :* TOKENSTRING CLOB DEFAULT ‘{}’ NOT NULL

*IBM Db2:* TOKENSTRING VARCHAR(20000) NOT NULL

*MS SQL:* TOKENSTRING NVARCHAR(MAX) NOT NULL


---+++ Oracle

To configure JAS to use Oracle, start by creating the required tables in your Oracle database.   You can use the sample [[https://jazz.net/wiki/pub/Deployment/PerformanceClusteredJAS/createOauthOracle.sql][createOauthOracle.sql]] as a starting point.   You will need to customize this sample for your environment (e.g. your passwords, paths, tablespaces).

createOauthOracle.sql creates two tablespaces:   OAUTH and OAUTH_TMP, and creates the tables in the OAUTH tablespace.   If you have an existing tablespace that you want to use, you can change the "CREATE TABLE" statements to use your existing tablespace.    If you want to create new tablespaces, then you should edit the DATAFILE and TEMPFILE clauses to point at the directory that you want to use for the tablespace.

The Jazz Authorization server expects to find a user named OAUTHDBSCHEMA, and it expects to find tables in a schema named OAUTHDBSCHEMA.    You will need to create the OAUTHDBSCHEMA user in Oracle.    Customize the "CREATE USER" statement to specify a password for the OAUTHDBSCHEMA user.  When you create the OAUTHDBSCHEMA user, you should also associate that user with the tablespace you plan to use for the Oauth tables.

After you finish customizing createOauthOracle.sql, you can log into your Oracle server and run the SQL script from sqlplus:

To start sqlplus:   sqlplus '/ as sysdba'

At the sqlplus prompt, run the SQL command by entering:    <verbatim>@createOauthOracle.sql</verbatim>

Note that the SQL for Oracle differs slightly from the SQL for Db2.    
   * The Db2 BIGINT data type is replaced with Oracle's NUMBER(19,0) 
   * DEFAULT keyword on CLOB fields must come before NOT NULL
   * Oracle uses additional clauses (TABLESPACE OAUTH STORAGE(INITIAL 50K)) for table creation
   * GRANT clauses in Oracle do not use the USER keyword
   * Enabling a Jazz server to use JAS requires that the DISPLAYNAME field support NULLs (so, NOT NULL keywords removed from that field)
   
After you create the tables, you must then configure all JAS instances to use the tables.    You do this by creating a data source definition in the [[https://jazz.net/wiki/pub/Deployment/PerformanceClusteredJAS/appConfigOracle.xml][appConfig.xml]] file (that file will be in the configuration folder on your JAS server).    You can refer to the attached appConfig.xml file for an example.   You will need to customize this to work with your specific Oracle server, but the section that defines a data source for Oracle is:

<verbatim>
<jdbcDriver id="oracle" libraryRef="ORAJCC4LIB"/>
<library id="ORAJCC4LIB" filesetRef="orajcc4" apiTypeVisibility="spec,ibm-api,third-party"/>
   <fileset dir="${shared.config.dir}/lib/global" id="orajcc4" includes="ojdbc6.jar"/>
<dataSource id="OAUTH2ORA" jndiName="jdbc/oauthProvider" jdbcDriverRef="oracle">
  <properties.oracle password="******" databaseName="ORCL" user="OAUTHDBSCHEMA" portNumber="1521" serverName="yourOracle.com"/>
</dataSource>
</verbatim>

Notes:
   * You need to copy the Oracle JBDC driver (ojdbc6.jar) from your Oracle server to the lib/global folder on your JAS server.    On Unix, that folder would be /opt/IBM/JazzAuthServer/wlp/usr/shared/config/lib/global, if you install JAS using the defaults.
   * In the "dataSource" record, 
      * For "password", using the password you specified when creating the OAUTHDBSCHEMA user in Oracle
      * For databaseName, use the Oracle SID for your database.   The sample uses the default value assigned by Oracle - ORCL.   Your value may be different.
      * Set the portNumber field to the port on which your Oracle instance is running.   The Oracle default is 1521;   your value may be different.
      * For serverName, enter the host name of the Oracle server.

You'll also need to update the "oauthProvider" record in appConfig.xml so that it specifies the Oracle data source (rather than the default Derby data source).   The "databaseStore" record should specify the name of the Oracle data source - in this example, "OAUTH2ORA".

<verbatim>
<oauthProvider id="JazzOP" authorizationGrantLifetime="604801" accessTokenLifetime="7201" customLoginURL="/jazzop/form/login" autoAuthorize="true" httpsRequired="true">
   <autoAuthorizeClient>client01</autoAuthorizeClient>
   <databaseStore dataSourceRef="OAUTH2ORA"/>
   </oauthProvider>
</verbatim>


---+++ Db2

To configure JAS to use Db2, start by creating the required tables in your Db2 database.   You can use the sample [[https://jazz.net/wiki/pub/Deployment/PerformanceClusteredJAS/createOauthTablesDB2.sql][createOauthTablesDB2.sql]] as a starting point.   You will need to customize this sample for your environment.

The script will create a database called "oauth2db", and then create tables in the schema "OAuthDBSchema".    You can use a different database name if you wish, but you must use the schema "OAuthDBSchema".   The tables require a buffer pool that has a page size of at least 8K.    The SQL script creates the necessary buffer pool and table spaces -  but you can skip this if your buffer pools use page sizes of 8K or greater already.

After you create the tables, you must then configure all JAS instances to use the tables.    You do this by creating a data source definition in the [[https://jazz.net/wiki/pub/Deployment/PerformanceClusteredJAS/appConfigDB2.xml][appConfig.xml]] file (that file will be in the configuration folder on your JAS server).    You can refer to the attached appConfigDB2.xml file for an example.   You will need to customize this to work with your specific Db2 server, but the section that defines a data source for Db2 is:

<verbatim>
<jdbcDriver id="db2Universal" libraryRef="DB2JCC4LIB"/>
<library id="DB2JCC4LIB" filesetRef="db2jcc4" apiTypeVisibility="spec,ibm-api,third-party"/>
<fileset dir="${shared.config.dir}/lib/global" id="db2jcc4" includes="db2jcc4.jar db2jcc_license_cu.jar"/>
<dataSource id="OAUTH2DBDS" jndiName="jdbc/oauthProvider" jdbcDriverRef="db2Universal">
<properties.db2.jcc password="*****" databaseName="OAUTH2DB" user="db2inst1" portNumber="50000" serverName="yourDb2.com" driverType="4"/>
</dataSource>
</verbatim>

Notes:
   * You need to copy the Db2 JBDC drivers (db2jcc4.jar, db2jcc_license_cu.jar) from your Db2 server to the lib/global folder on your JAS server.    On Unix, that folder would be /opt/IBM/JazzAuthServer/wlp/usr/shared/config/lib/global, if you install JAS using the defaults.
   * In the "dataSource" record, 
      * For "user", provide the name of a Db2 user to be used for connecting to the database 
      * For "password", using the password for the Db2 specified above
      * For databaseName, use the name of the database you created previously ("OAUTH2DB" if you used the sample script)
      * Set the portNumber field to the port on which your Db2 instance is running.   The default is 50000;   your value may be different.
      * For serverName, enter the host name of the Db2 server.

You'll also need to update the "oauthProvider" record in appConfig.xml so that it specifies the Db2 data source (rather than the default Derby data source).   The "databaseStore" record should specify the name of the Db2 data source - in this example, "OAUTH2DBDS".

<verbatim>
<oauthProvider id="JazzOP" authorizationGrantLifetime="604801" accessTokenLifetime="7201" customLoginURL="/jazzop/form/login" autoAuthorize="true" httpsRequired="true">
<autoAuthorizeClient>client01</autoAuthorizeClient>
<databaseStore dataSourceRef="OAUTH2DBDS"/>
</oauthProvider>
</verbatim>

---+++ Microsoft SQL Server

*Note:* It is applicable up to and including 7.0.2

To configure JAS to use Microsoft SQL Server, start by creating the required tables in your SQL Server database.   You can use the sample [[https://jazz.net/wiki/pub/Deployment/PerformanceClusteredJAS/createOauthMSSQLv2.sql][createOauthMSSQL.sql]] as a starting point.   You will need to customize this sample for your environment.

The script will create a database called "oauth2db", and then create tables in the schema "OAuthDBSchema".    You can use a different database name if you wish, but you must use the schema "OAuthDBSchema".   The tables require a buffer pool that has a page size of at least 8K.    The SQL script creates the necessary buffer pool and table spaces -  but you can skip this if your buffer pools use page sizes of 8K or greater already.

<verbatim>
<oauthProvider id="JazzOP"
    httpsRequired="true"
    autoAuthorize="true"
    customLoginURL="/jazzop/form/login" 
    accessTokenLifetime="7201" 
    authorizationGrantLifetime="604801">
<autoAuthorizeClient>client01</autoAuthorizeClient>
  <databaseStore dataSourceRef="OAUTH2DBDS" /> 
</oauthProvider>
</verbatim>

You'll also need to update the "oauthProvider" record in appConfig.xml so that it specifies the MS SQL Server data source (rather than the default Derby data source).   The "databaseStore" record should specify the name of the SQL Server data source - in this example, "OAUTH2DBDS".

<verbatim>
<dataSource id="OAUTH2DBDS" jndiName="jdbc/OAUTH2DB">
    <jdbcDriver libraryRef="MSJDBCLib"/>
    <properties.microsoft.sqlserver user="username" password="Password" databaseName="OAUTH2DB" serverName="SQLDBServer" portNumber="1433"/>
</dataSource>
<library id="MSJDBCLib">
     <fileset dir="${shared.config.dir}/lib/global" id="sqljdbc41" includes="sqljdbc41.jar"/>
</library>
</verbatim>

Notes:
   * You need to copy the Microsoft SQL Server JBDC drivers (sqljdbc41.jar) from your MS SQL server to the lib/global folder on your JAS server.    On Unix, that folder would be /opt/IBM/JazzAuthServer/wlp/usr/shared/config/lib/global, if you install JAS using the defaults.
   * As per MS [[https://technet.microsoft.com/en-us/library/ms186939(v=sql.110).aspx][Documentation]] The maximum number of characters allowed by sql server is 4,000 or we can use MAX which supports 2GB storage in size.
   * In the "dataSource" record, 
      * For "user", provide the name of a MS SQL Server user to be used for connecting to the database 
      * For "password", using the password for the SQL Server specified above
      * For databaseName, use the name of the database you created previously ("OAUTH2DB" if you used the sample script)
      * Set the portNumber field to the port on which your MS SQL Server instance is running.   The default is 1433;   your value may be different.
      * For serverName, enter the host name of the Microsoft SQL Server.

---++ Setting up dispatching using IBM HTTP Server

The Jazz Authorization Server is really just a Websphere Liberty server.   The method of using IHS with Liberty can be found in the Liberty Infocenter - see [[http://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/twlp_admin_webserver_plugin.html?cp=SSAW57_8.5.5][Configuring a web server plug-in for Liberty]].   You can follow the basic steps there to create a plugin for Liberty using the Websphere Customization Toolbox (WCT).   Once you have a basic plugin, you can customize it to dispatch traffic across your JAS instances.

The additional steps involved to get IHS to dispatch traffic across multiple JAS instances are:
   * Add a clone id  value to the server.xml file for each JAS server
   * Modify the "ServerCluster" section in the plug-in.xml file on the IHS server, to route traffic to the available JAS servers
   * Configure the SSL truststores and keystores on IHS and the JAS instances

---+++ Updating server.xml on each JAS instance

Add a line similar to the following to the server.xml file on each JAS instance:

<verbatim>
<httpSession cookieHttpOnly="false" cloneId="cloneid00001"/>
</verbatim>

Assign each JAS server a different id (e.g. cloneid00001, cloneid00002), which guarantees that each id is unique in the cluster.   This is used by IHS to implement session affinity.

See the attached [[https://jazz.net/wiki/pub/Deployment/PerformanceClusteredJAS/server.xml][server.xml]] for an example.

---+++ Updating the Websphere plugin for dispatching

Once you've created a basic WebSphere plugin XML file, you can manually update it to route traffic across your JAS instances.    You need to make the following changes:
   * Add "VirtualHost" entries for the JAS SSL and non-SSL ports (9280 and 9643 if you are using the defaults)
   * Modify the "ServerCluster" entry to group each of your JAS instances together into a single cluster
   * Update the URI group to include the JAS URIs that the cluster will handle
   
The attached [[https://jazz.net/wiki/pub/Deployment/PerformanceClusteredJAS/plugin-cfg.xml][plugin-cfg.xml]] file can be used as an template - you'll need to modify this for your own deployment.   In this example, there is a server cluster called "server1_Cluster", and this routes traffic to two JAS instances - one called vtrjas2.rtp.raleigh.ibm.com, and one called vtrjas3.rtp.raleigh.ibm.com).    Both JAS instances are configured with the default ports (9643 for SSL traffic, and 9280 for non-SSL traffic).

The uri group for JAS should look like this:

<verbatim>
    <UriGroup Name="server1_Cluster_URIs">
        <Uri Name="/j_security_check"/>
        <Uri Name="/ibm_security_logout"/>
        <Uri Name="/jazzop/form/login"/>
        <Uri Name="/jazzop/*"/>
        <Uri Name="/ibm/*"/>
        <Uri Name="/oidc/*"/>
    </UriGroup>
</verbatim>

If you are using the default names for ServerCluster, UriGroup, and VirtualHostGroup, the route for the cluster should look like this:

<verbatim>
    <Route ServerCluster="server1_Cluster"
        UriGroup="server1_Cluster_URIs" VirtualHostGroup="default_host"/>
</verbatim>

The virtual host group would be something like this.   The JAS ports  (9643, 9280) must appear here, along with any other ports on which IHS is listening.

<verbatim>
    <VirtualHostGroup Name="default_host">
        <VirtualHost Name="*:9280"/>
        <VirtualHost Name="*:80"/>
        <VirtualHost Name="*:443"/>
        <VirtualHost Name="*:9643"/>
     </VirtualHostGroup>
</verbatim>

Finally, the server cluster entry would be similar to the entry below.   You'll need to adjust this based on the number of JAS instances you have, which clone ids you assigned to each instance, and what your JAS ports are.  Make sure the attribute *CloneID* is present for each *Server* element - its value is the same with the cloneId specified in the server.xml for each JAS instance accordingly.  You may manually add this attribute or change value for it.  Please note that in the sample code below, we are using a keystore called "ihskeys.kdb" for the plugins, the creation and configuration of this keystore will be covered in the section [[#Configuring_SSL_keystores_for_JA][Configuring SSL keystores for JAS/IHS communication]].

<verbatim>
    <ServerCluster Name="server1_Cluster" CloneSeparatorChange="false" GetDWLMTable="false"
        LoadBalance="Round Robin" PostBufferSize="64" IgnoreAffinityRequests="true"
        PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60">
        <Server Name="server1" CloneID="cloneid00001" ConnectTimeout="5" ExtendedHandshake="false" ServerIOTimeout="60" LoadBalanceWeight="1" MaxConnections="-1" WaitForContinue="false">
            <Transport Hostname="vtrjas2.rtp.raleigh.ibm.com" Port="9280" Protocol="http"/>
            <Transport Hostname="vtrjas2.rtp.raleigh.ibm.com" Port="9643" Protocol="https">
                <Property name="keyring" value="/opt/IBM/HTTPServer/bin/ihskeys.kdb"/>
                <Property name="stashfile" value="/opt/IBM/HTTPServer/bin/ihskeys.sth"/>
            </Transport>
        </Server>
        <Server Name="server2" CloneID="cloneid00002" ConnectTimeout="5" ExtendedHandshake="false" ServerIOTimeout="60" LoadBalanceWeight="1" MaxConnections="-1" WaitForContinue="false">
            <Transport Hostname="vtrjas3.rtp.raleigh.ibm.com" Port="9280" Protocol="http"/>
            <Transport Hostname="vtrjas3.rtp.raleigh.ibm.com" Port="9643" Protocol="https">
                <Property name="keyring" value="/opt/IBM/HTTPServer/bin/ihskeys.kdb"/>
                <Property name="stashfile" value="/opt/IBM/HTTPServer/bin/ihskeys.sth"/>
            </Transport>
        </Server>
        <PrimaryServers>
            <Server Name="server1"/>
            <Server Name="server2"/>
        </PrimaryServers>
    </ServerCluster>
</verbatim>

---+++ Configuring SSL keystores for JAS/IHS communication

To allow the IHS and JAS instances to communicate over SSL, you'll need to set up trust and key stores on IHS and JAS, and then exchange certificates.   There are several useful references in the documentation:
   * [[https://jazz.net/help-dev/clm/index.jsp?topic=%2Fcom.ibm.jazz.install.doc%2Ftopics%2Ft_jsasso_jas_cli_certif_conf.html][Configuring security certificates]]
   * [[https://jazz.net/help-dev/clm/index.jsp?topic=%2Fcom.ibm.jazz.install.doc%2Ftopics%2Ft_jsasso_jas_cli_certif_conf.html][Installing a security certificate]]
   * [[https://jazz.net/help-dev/clm/index.jsp?topic=%2Fcom.ibm.jazz.install.doc%2Ftopics%2Ft_jsasso_jas_cli_certif_conf.html][Enabling SSL communication in Liberty]]
   * [[https://www.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/rwlp_ssl.html][Liberty - SSL configuration attributes]]
   * [[https://www.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/rwlp_sec_keystores.html][Liberty:  Keystores]]

For our test configuration, we generated new key and trust stores using self-signed certificates.    The steps we followed are below for Linux;   you may need to adjust these slightly if you already have key stores, or if you want to use a signed certificate, or if you are working on Windows.

*Step 1:   Create a new key store for the IHS server*

On the IHS machine, cd to <IHS_INSTALL_DIR>/bin, e.g. /opt/IBM/HTTPServer/bin, and run the following commands 
<verbatim>
./gskcmd -keydb -create -db ihskeys.kdb -pw xxxxx -expire 3650 -stash -type cms
./gskcmd -cert -create -db ihskeys.kdb  -label default -expire 3650 -size 2048 -dn "CN=xxxxx" -default_cert yes -pw xxxxx
</verbatim>
, where *dn* denotes the Distinguished Name for the IHS server, use a fully qualified name for CN. For example -dn "CN=yourhostname.yourdomain.com".


*Step 2:  Create key and trust stores on all JAS instances;   export the public key*

The commands below should be executed on each JAS instance in the cluster. Take a note for the values of *alias*, *keystore*, *storepass*, and *storetype*; these will be applied to the appConfig.xml configuration file for JAS.

<verbatim>
keytool -genkeypair -alias default -keystore jas.p12 -storepass xxxxx -validity 3650 -keyalg RSA -keysize 2048 -storetype PKCS12
keytool -export -alias default -keystore jas.p12 -storetype pkcs12 -file jas1pub.arm -storepass xxxxx
keytool -import -alias default -keystore jastrust.p12 -storepass xxxxx -file jas1pub.arm -storetype PKCS12
</verbatim>

Copy the exported key to your IHS server.   Use a unique file name in the -file option on each JAS server.   Here, we use a file "jas1pub.arm" for the JAS instance JAS1.

*Step 3:   Add the public key to the IHS key store*

Start the ikeyman tool and open your IHS keystore.   Select "Signer Certificates", and then select "Add..." for each of the JAS public keys exported in step 2.
 <img src="https://jazz.net/wiki/pub/Deployment/PerformanceClusteredJAS/IHSSigner.png" alt="IHSSigner.png"> 


*Step 4:   Export the IHS certificate*

On your IHS server, start the ikeyman tool.   Open your IHS keystore.    Select "Personal certificates", then "Extract Certificate..." ; save the certificate as "ihscert.arm".

 <img src="https://jazz.net/wiki/pub/Deployment/PerformanceClusteredJAS/IHSCert.png" alt="IHSCert.png"> 

*Step 5:   Import the IHS certificate into the JAS key stores*

Copy ihscert.arm (generated in step 4) to each of your JAS instances.   Then, import the certificate into the trust store by running the following command:

<verbatim>
keytool -import -alias ihsdefault -keystore jastrust.p12 -storepass xxxxx -trustcacerts -file ihscert.arm -storetype PKCS12
</verbatim>

*Step 6:   Update appConfig.xml on the JAS instances to use the key and trust stores*

Update appConfig.xml with a section similar to this:

<verbatim>
    <keyStore id="defaultKeyStore"
     location="jas.p12"
     type="PKCS12" password="xxxxx" />

    <keyStore id="defaultTrustStore"
     location="jastrust.p12"
     type="PKCS12" password="xxxxx" />

    <ssl id="defaultSSLConfig"
     keyStoreRef="defaultKeyStore"
     trustStoreRef="defaultTrustStore"
     serverKeyAlias="default"
     clientAuthenticationSupported="true"/>
</verbatim>
, where serverKayAlias is the value of *alias* you used in Step 2 above. Make sure the location to the keyStores (in the example above, "jas.p12" and "jastrust.p12") are using absolute directory path, or they have to exist in the same directory of appConfig.xml.


You can refer to the attached appConfig.xml sample file as well.

After the configurations are completed and servers restarted, the clustered JAS can be accessed from https://your_IHS_server:your_IHS_port/a_uri, for instance you can verify that the JAS server is online via https://your_IHS_server/jazzop/form/login

---++ Configuring Jazz servers to use a JAS cluster

Once you have configured an IHS to serve as a network dispatcher for JAS instances, you can configure your Jazz servers to use the JAS cluster.    Use the host name for the IHS server when prompted for the host name of the JAS server.    Jazz servers will then direct login requests to your IHS, and your IHS will redirect the requests to one of your JAS instances.

If you are installing a Jazz server for the first time and you have already set up your JAS cluster, you should select the option "Enable Jazz Security Architecture SSO" in Installation Manager.

 <img src="https://jazz.net/wiki/pub/Deployment/PerformanceClusteredJAS/IMsso.png" alt="IMsso.png"> 

Then, when running Jazz setup, you'll be prompted to enter the host name for the Jazz Authorization Server.   Enter the name of your IHS server here.

 <img src="https://jazz.net/wiki/pub/Deployment/PerformanceClusteredJAS/jtsSetup.png" alt="jtsSetup.png">

If you have an existing Jazz server that is not already JAS-enabled, you can [[http://www.ibm.com/support/knowledgecenter/SSYMRC_6.0.2/com.ibm.jazz.install.doc/topics/r_repotools_migrateToJsaSso.html][follow the steps in the Infocenter]] to enable it.

---++ Scalability and high availability testing

We tested a clustered JAS system to understand its scalability limits.    We used a single IBM HTTP Server with two Jazz Authorization servers, and used a DB2 server to store the JAS configuration databases.    The IHS and JAS servers were virtual machines running Linux Red hat 6, and each system had 4 virtual CPUs and 8G of RAM.    The database server was a physical system with 24 logical processors and 32G of RAM.  We had basic tuning done on the IHS server.

This cluster was able to process 220 logins per second.    This is roughly equivalent to a user population of 200,000 logging in over 15 minutes.     This is an extreme simulation of what might happen if everyone in an organization arrived at work at around the same time, and logged into their Jazz systems immediately.    The JAS cluster can easily handle this extreme load, and in fact, the limit we found was due to a configuration error on one of the JAS instances (we had not increased the ulimit settings for open files).   

We also ran a failover simulation at scale.   We placed the JAS cluster under constant login load, and then took down one of the JAS instances.   As expected, IHS detected the failure and routed traffic to the other JAS node.    There was one brief failure in the client simulation at this point - there was a login operation in progress when the JAS instance was shut down, and this interrupted operation resulted in an HTTP 500 error at the client.   But the workload immediately recovered.    When the failed JAS system was brought back up, IHS detected this and began to route traffic across both JAS instances.    

---++ Limitations

The approach described in this article applies only when setting up a new Jazz Authorization server.    If you have already set up a Jazz Authorization server and you've set up a Jazz deployment to use it, you can't create a cluster.     

There are two reasons for this:
   * If you have an existing Jazz Authorization Server, it will most likely have been configured to use the default Derby database.     You can't easily transfer the configuration data to a shared database server.   JAS clustering requires a way to share configuration data amongst cluster nodes.
   * Once you have configured a Jazz deployment to use a particular Jazz Authorization Server, you can't change it to use a new Jazz Authorization Server.    So, you are not able to set up a new JAS cluster and reconfigure your existing Jazz deployment.    

---++ Tuning

There are a few tuning options available for the Jazz Authorization server.    These may be useful if you see poor login performance in the CLM applications, or if you see signs of performance issues on the Jazz Authorization Server (high web container counts, error messages).

If you are using the SCIM feature for JAS, you will need to add the following line to server.xml (see maxSearchResults to a value greater than the number of users in your LDAP repository).   Symptoms:   You will see errors during jts setup, and you'll also see errors on JAS in messages.log.

<verbatim>
<federatedRepository maxSearchResults="80000"/> 
</verbatim>

For larger concurrent user loads, you may need to increase the size of the database connection pool (the default is 50).   Add the following line to server.xml:

<verbatim>
<connectionManager purgePolicy="FailingConnectionOnly" maxPoolSize="200" />
</verbatim>

To debug possible performance issues related to database access, you can enable timed operation monitoring on JAS.     Add the following line to the featureManager section in server.xml.     This will generate a report of the top 10 SQL calls to messages.log (and it will also generate a message when SQL calls take longer than expected).     

<verbatim>
<feature>timedOperations-1.0</feature>
</verbatim>

You can control the frequency of the reporting by adding the following line to server.xml (reportFrequency is in hours):

<verbatim>
<timedOperation enableReport="true" reportFrequency="1" />
</verbatim>

If you see messages in the JAS messages.log file related to the number of expired tokens, you can adjust some tuning parameters in appConfig.xml to reduce the number of entries in the OAUTH20CACHE table.    The symptom is a message like this:

<verbatim>
[3/31/17 12:59:12:282 EDT] 00007e46 om.ibm.ws.security.oauth20.plugins.db.CachedDBOidcTokenStore W 109496 expired tokens to delete. Consider increasing OAuth provider cleanup interval
</verbatim>

You can increase the frequency at which expired tokens are removed by adding this to the databaseStore entry in appConfig.xml:

<verbatim>
<databaseStore dataSourceRef="OAUTH2DBDS" cleanupExpiredTokenInterval="600s" />
</verbatim>

You can adjust the lifetime of the cached access tokens by changing accessTokenLifetime and authorizationGrantLifetime (in the oauthProvider section of appConfig.xml).    By default, the accessTokenLifetime is 2 hours, and the authorizationGrantLifetime is 7 days.     You can reduce authorizationGrantLifetime to 1 day.

You may be able to improve performance by adjusting the isolation level of the JDBC driver.    You can do this by adding the following entry to the dataSource section of appConfig.xml:

<verbatim>
isolationLevel="TRANSACTION_READ_COMMITTED"
</verbatim>

---+++++!! Additional notes:

   * The scripts for creating tables are for example only. You should change them according to your usage and environment. For example, if the TOKENSTRING value of 2048 is too small, you can increase that by using a datatype of TOKENSTRING NVARCHAR(MAX) NOT NULL,.
   * Please review the Interactive Installation Guide for specific details on setting up the Authorization Server for the version of the product you are installing.
   * [[https://www.ibm.com/support/knowledgecenter/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/cwlp_oauth_sql.html][WebSphere Liberty: Configuring Persistent OAUTH Services]]

---+++++!! Related topics: 

   * [[http://www-01.ibm.com/support/docview.wss?uid=swg27020055&aid=1][Websphere plugin and session affinity]]
   * [[https://jazz.net/library/article/75][Jazz server authentication explained]]
   * [[https://www.ibm.com/support/knowledgecenter/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/cwlp_timeop.html][Liberty:   Timed operations and JDBC calls]]
   * [[https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.0.0/com.ibm.websphere.nd.doc/info/ae/ae/cwbs_oauthdefineprovider.html][OAuth configuration parameters in Liberty]]
   * [[https://www.ibm.com/support/knowledgecenter/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/twlp_tun.html][Tuning options for Liberty]]


---+++++!! Additional contributors: Main.KrishnaKanthNaik, Main.ShubjitNaik

<sticky></div></sticky>
Attachments
Topic attachments
I	Attachment	Action	Size	Date	Who	Comment
png	IHSCert.png	manage	154.0 K	2016-07-11 - 17:26	VaughnRokosz
png	IHSSigner.png	manage	18.7 K	2016-07-11 - 17:05	VaughnRokosz
png	IMsso.png	manage	221.8 K	2016-07-09 - 14:57	VaughnRokosz
png	Overview.png	manage	12.1 K	2016-07-08 - 13:52	VaughnRokosz
xml	appConfigDB2.xml	manage	6.2 K	2016-07-08 - 17:44	VaughnRokosz
xml	appConfigOracle.xml	manage	6.3 K	2016-07-08 - 15:01	VaughnRokosz
sql	createOauthMSSQL.sql	manage	2.1 K	2019-09-06 - 11:18	ShubjitNaik	Updated NVARCHAR(2048) to NVARCHAR(MAX)
sql	createOauthMSSQLv2.sql	manage	2.1 K	2019-09-06 - 11:12	ShubjitNaik
sql	createOauthOracle.sql	manage	2.5 K	2016-07-08 - 15:01	VaughnRokosz
sql	createOauthTablesDB2.sql	manage	2.1 K	2016-07-08 - 17:43	VaughnRokosz
png	jtsSetup.png	manage	66.7 K	2016-07-09 - 14:57	VaughnRokosz
xml	plugin-cfg.xml	manage	3.3 K	2016-07-12 - 13:04	UnknownUser
xml	server.xml	manage	1.7 K	2016-07-08 - 19:27	VaughnRokosz
Deployment
Community information and contribution guidelines
Status icon key:
To do
Under construction
New
Updated
Constant change
None - stable page
Smaller versions of status icons for inline text:
Copyright © by IBM and non-IBM contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.