Configuring TLS Extended Master Secret for WebSphere Application Server and WebSphere Liberty

Authors: HongyanHuo SentellaCystrunk Contributors: JonAgnew AlexBernstein KelvinLow SkyeBischoff
Build basis: All CLM Products 6.0.6 GA

Introduction

The 6.0.6 release of IBM® Rational® Collaborative Lifecycle Management (CLM) products now include Java JDK level v8 SR5 FP11 (8.0.5.11) in Liberty and we encourage upgrading to at least Java JDK level 8.0.5.10 or 7.1.4.20 for WebSphere server products. With this set of Java upgrades comes a new Java Security requirement 1.2 TLS protocol jdk.tls.useExtendedMasterSecret (EMS). As outlined in this article Java Extended Master Secret support requires a full handshake between every SSL/TLS connection between Liberty and the proxy servers. Unfortunately we have found not all Proxy/HTTP servers support support EMS at this time. In situations where EMS support is not present, enabling EMS can result in performance problems with your CLM products hosted by Liberty or WebSphere. This article outlines what environments can be impacted by this problem, how to diagnosis if your environment is impacted, and what solutions we recommend for configuring your server to resolve the issue.

Impacted deployment

Liberty/!WebSphere with JDK level v8 SR5 FP10 (8.0.5.10), 7 SR10 FP20 (7.0.10.20), or 7 R1 SR4 FP20 (7.1.4.20) and above[1], with a proxy/web server that does not support or enable EMS (aka Extended Master Secret) but connects directly to Liberty/!WebSphere with SSL. For example, the default install of Liberty 18.0.0.1 with one of the following

• Haproxy 1.5.8/!OpenSSL 1.0.2: Haproxy is configured as SSL termination and connects to backend Liberty servers using SSL only
• A legacy web server with underlying SSL libraries and/or SSL modules that are not EMS compatible: depending on the SSL configuration with backend Liberty servers

Symptoms

After CLM 6.0.6 Liberty upgrade or Java upgrade on any platform:

• Slow page loading and/or timeouts
• Significantly decreased server throughput
• Increased server CPU utilization on the Liberty/!WebSphere server
• When SSL tracing is enabled on the Liberty/!WebSphere server, excessive messages such as "%% Initialized: [Session-, SSL_NULL_WITH_NULL_NULL]", "%% Negotiating: [Session-, ]" are shown:

Cause

Full handshake (CPU expensive) is required for every single SSL/TLS connection between Liberty and the proxy servers, when the new EMS security feature is enabled by default on the Liberty server's JVM, while the proxy server is not. Details are in IBM ARAR IJ05598 at https://www-01.ibm.com/support/docview.wss?uid=swg1IJ05598

Solution

Option 1a Add Liberty JVM Arg

The simplest solution for Liberty is to add a JVM argument to the server startup script to disable EMS. In start-jazz on JAS liberty server, you add the following to the existing args:

export JVM_ARGS="... -Djdk.tls.useExtendedMasterSecret=false"

In server.startup on remaining CLM liberty servers, you add the following

JAVA_OPTS="$JAVA_OPTS -Djdk.tls.useExtendedMasterSecret=false"

Option 1b Add WebSphere JVM Arg

The simplest solution for WebSphere is to add a JVM argument to the servers Generic JVM Arguments. Go to Application servers > server1 > Process definition > Java Virtual Machine

Add "-Djdk.tls.useExtendedMasterSecret=false" to the Generic JVM Arguments

Option 2 - Upgrade Proxy

For Liberty or WebSphere you may choose to leave jdk.tls.useExtendedMasterSecret=true (the default) and instead ensure your front end Proxy/HTTP server is up to date with the newest SSL libraries and Java. For the WebSphere IHS or Proxy server you would need to upgrade to the newest 8.x or 9.x version matching your WebSphere Application Server, and upgrade your Java JDK on both your IHS/Proxy-server and WebSphere Application Server. For other solutions like Haproxy you would have to download and install the newest version of OpenSSL, or wait for an updated Haproxy for your Linux distribution. Other proxy solutions will likewise require updates to their core product or underlining SSL libraries.

Diagnosis

Option 1 - Support Check

To demonstrate the change in behavior without enabling Java SSL debugging, you can run openssl s_client with the -reconnect option, using a version of openssl that does not support the Extended Master Secret extension. For example install an older version of OpenSSL such as 1.0.2k-fips from 26 Jan 2017.

$ openssl version
!OpenSSL 1.0.2k-fips  26 Jan 2017

# Against the old java, or a new java with -Djdk.tls.useExtendedMasterSecret=false , the session id can be reused to resume the session.
$ openssl s_client -reconnect -connect jontestvm.rtp.raleigh.ibm.com:9643 < /dev/null 2>&1 | grep 'Cipher is'
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Reused, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Reused, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Reused, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Reused, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Reused, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384

# Against a new java which supports the Extended Master Secret extension, the server rejects the client's attempt to resume using the session id, forcing a full handshake each time.
$ openssl s_client -reconnect -connect jontestvm.rtp.raleigh.ibm.com:9644 < /dev/null 2>&1 | grep 'Cipher is'
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384

Option 2 - Debug Check

When SSL tracing or SSL debugging is enabled: in every ClientHello initiated by proxy server, an attribute called "Extension extended_master_secret" is NOT present, followed by messages indicating new key generation and negotiation start and complete before ServerHello. To enable SSL debugging, you can set the JVM arg, -Djavax.net.debug=ssl. Start the server and observe the log:

If excessive handshaking is occurring you will see repetitive messages like the following and no Extension extended_master_secret notification:

*** ClientHello, TLSv1.2
...
***
%% Initialized:  [Session-10387, SSL_NULL_WITH_NULL_NULL]
ssl: ServerHandshaker.setupPrivateKeyAndChain RSA
matching alias: default
ssl: ServerHandshaker.setupPrivateKeyAndChain, chooseEngineServerAlias default
ssl: ServerHandshaker.setupPrivateKeyAndChain, return true
JsseJCE:  Using KeyPairGenerator EC from provider TBD via init
JsseJCE:  Using SecureRandom SHA2DRBG from provider IBMJCE version 1.8
JsseJCE:  Using KeyPairGenerator EC from provider TBD via init
ECDHCrypt:  ECDH KeyPairGenerator  from provider from init IBMJCE version 1.8
%% Negotiating:  [Session-10387, SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA256]
JsseJCE:  Using MessageDigest SHA-256 from provider IBMJCE version 1.8
*** ServerHello, TLSv1.2

If the configuration is correct with the newer JDK , after the first full handshake is made you will see the Extension extended_master_secret notification present:

*** ClientHello, TLSv1.2
...
Compression Methods:  {
0
}
Extension elliptic_curves, curve names: {secp256r1, secp384r1, secp521r1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA224withECDSA, SHA224withRSA, SHA1withECDSA, SHA1withRSA, SHA256withDSA, SHA1withDSA
Extension extended_master_secret
Extension server_name, server_name: [type=host_name (0), value=v-6c26c0cb-ihs.rtp.raleigh.ibm.com]
***
qm: AsynchronousTaskRunner-3 @@ 09:35, WRITE: TLSv1.2 Handshake, length = 260
qm: AsynchronousTaskRunner-3 @@ 09:35, READ: TLSv1.2 Handshake, length = 91
*** ServerHello, TLSv1.2
RandomCookie:
...

Note: If EMS is disabled, the extension will not shown regardless of the SSL configuration.