Question on SSL certificates and Jazz proxies; single WAS server with three profiles and five DNS aliases
I am helping a consultant try to identify the correct certificate configuration for this setup:
They have three applications, each in their own WAS instance, running on the same host server.example.com under three different server names. These are the public URIs:
https://jts.example.com:9443/jts
https://rtc.example.com:9444/ccm
https://rqm.example.com:9445/qm
End users can also connect in on a rewriting proxy, which is not supported but works. This is not part of the problem:
https://external.example.com/ccm
Everything works well in this environment, except one thing: "Proxy" invocations between CLM applications take over two minutes to complete.
For example, when you expand the "Home" or "Logout" menu on RTC or RQM, it takes two minutes to complete. 120s.
For example the logout menu from jts/admin invokes this which loads instantly:
https://jts.example.com:9443/jts/users/erikm/smallPreview
But the logout menu from ccm/admin loads this which takes more than two minutes to complete:
https://rtc.example.com:9445/ccm/proxy?uri=https%3A%2F%2Fjts.example.com%3A9443%2Fjts%2Fusers%2Ferikm%2FsmallPreview
From experience I know this indicates a SOAP problem. Websphere application servers unable to share login data.
So I have started looking into WAS/ffdc logs to see what problems these servers have speaking to each other.
Indeed I see certificate errors:
Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: com.ibm.jsse2.util.m: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=jts.example.com, OU=Root Certificate, OU=jtsNode02Cell, OU=jtsNode02, O=IBM, C=US is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error; targetException=java.lang.IllegalArgumentException: Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: com.ibm.jsse2.util.m: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=jts.example.com, OU=Root Certificate, OU=jtsNode02Cell, OU=jtsNode02, O=IBM, C=US is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:475)
at org.apache.soap.rpc.Call.WASinvoke(Call.java:510)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient$4.run(SOAPConnectorClient.java:387)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:372)
... 65 more
This occurs for RTC and RQM with this message:
com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host jts.example.com at port 8880.
But it occurs for JTS with this message:
com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host jts.example.com at port 8881.
All in all, I wonder how to best help this consultant fix their certificates. I am considering this must gather, but they want to avoid a server restart: http://www-01.ibm.com/support/docview.wss?uid=swg21971762
Two observations that I may need to confirm:
1. RTC has a default certificate for jts.example.com. I suspect this could be because WAS for RTC was invoked/installed on jts.example.com instead of rtc.example.com
2. The difference in ports between JTS and CCM's port connectors could further indicate that CCM is configured with the wrong hostname.
Some observations on what the problem IS NOT:
* CCM is not totally unable to speak to JTS such as during user login, even though no certificates have been shared between servers.
* The certificate is reported as "not trusted", not "not existing". Not sure if this is significant.
They have three applications, each in their own WAS instance, running on the same host server.example.com under three different server names. These are the public URIs:
https://jts.example.com:9443/jts
https://rtc.example.com:9444/ccm
https://rqm.example.com:9445/qm
End users can also connect in on a rewriting proxy, which is not supported but works. This is not part of the problem:
https://external.example.com/ccm
Everything works well in this environment, except one thing: "Proxy" invocations between CLM applications take over two minutes to complete.
For example, when you expand the "Home" or "Logout" menu on RTC or RQM, it takes two minutes to complete. 120s.
For example the logout menu from jts/admin invokes this which loads instantly:
https://jts.example.com:9443/jts/users/erikm/smallPreview
But the logout menu from ccm/admin loads this which takes more than two minutes to complete:
https://rtc.example.com:9445/ccm/proxy?uri=https%3A%2F%2Fjts.example.com%3A9443%2Fjts%2Fusers%2Ferikm%2FsmallPreview
From experience I know this indicates a SOAP problem. Websphere application servers unable to share login data.
So I have started looking into WAS/ffdc logs to see what problems these servers have speaking to each other.
Indeed I see certificate errors:
Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: com.ibm.jsse2.util.m: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=jts.example.com, OU=Root Certificate, OU=jtsNode02Cell, OU=jtsNode02, O=IBM, C=US is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error; targetException=java.lang.IllegalArgumentException: Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: com.ibm.jsse2.util.m: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
java.security.cert.CertPathValidatorException: The certificate issued by CN=jts.example.com, OU=Root Certificate, OU=jtsNode02Cell, OU=jtsNode02, O=IBM, C=US is not trusted; internal cause is:
java.security.cert.CertPathValidatorException: Certificate chaining error]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:475)
at org.apache.soap.rpc.Call.WASinvoke(Call.java:510)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient$4.run(SOAPConnectorClient.java:387)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.reconnect(SOAPConnectorClient.java:372)
... 65 more
This occurs for RTC and RQM with this message:
com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host jts.example.com at port 8880.
But it occurs for JTS with this message:
com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host jts.example.com at port 8881.
All in all, I wonder how to best help this consultant fix their certificates. I am considering this must gather, but they want to avoid a server restart: http://www-01.ibm.com/support/docview.wss?uid=swg21971762
Two observations that I may need to confirm:
1. RTC has a default certificate for jts.example.com. I suspect this could be because WAS for RTC was invoked/installed on jts.example.com instead of rtc.example.com
2. The difference in ports between JTS and CCM's port connectors could further indicate that CCM is configured with the wrong hostname.
Some observations on what the problem IS NOT:
* CCM is not totally unable to speak to JTS such as during user login, even though no certificates have been shared between servers.
* The certificate is reported as "not trusted", not "not existing". Not sure if this is significant.
2 answers
Looks like the error is the same mentioned in this technote http://www-01.ibm.com/support/docview.wss?uid=swg21369939
The products mentioned in the technote are completely different, but I think the issue is the same (Certificate chaining errors between 2 applications), and the fix would be the same: verify that the truststore in one application contains the proper 'signer certificate' for the certificate chain provided by the other application.
The products mentioned in the technote are completely different, but I think the issue is the same (Certificate chaining errors between 2 applications), and the fix would be the same: verify that the truststore in one application contains the proper 'signer certificate' for the certificate chain provided by the other application.