Configuring Enterprise CLM Reverse Proxies: Apache and mod_proxy

Authors: SeanWilbur
Build basis: Collaborative Lifecycle Management Platform 3.0.1, Collaborative Lifecycle Management Platform 4.0.x

This page describes a usage or software configuration that is not supported by IBM.

This article outlines the process for configuring a reverse proxy to support a consistent Public URL1 in a flexible deployment topology. Provided here are a few sample deployment options to reference when configuring your own reverse proxy configuration.

Other topics in this series

• Understanding reverse proxy
• Configuring Enterprise CLM Reverse Proxies: WebSphere 8 and IHS 8
• Configuring Enterprise CLM Reverse Proxies: WebSphere 8.5 ND Proxy

Prerequisites

1. Apache httpd basic administrative skills are not covered here.
   
2. Write access Apache/IHS machine configuration directory:
   We will need to have write access to the location of httpd.conf(apache/conf) and the key.kdb/key.sth(few common options apache, apache/conf, /etc/ssl ) files are located.
   
3. Apache httpd server with mod_ssl & mod_proxy support:
   Next we need to ensure that the version of apache we have can support the modules we need. While both of these options are common it still requires the builder to select them as neither --enable-proxy or --enable-ssl are default compile options2. This is generally the case is most packaged versions of Apache today but something to check if you don't see the modules available in your modules directory.
   
4. Configure this external facing httpd server to support SSL:
   Configuring the external facing Apache server for SSL is required and well documented for both Apache and IHS.
   
5. Running Jazz deployment (configured or not):
   1. An existing configured environment that has already been setup with a defined Public URL and your configuring the proxy to allow for a distributed deployment3.
   2. A new install that will required the reverse proxy before you can complete /jts/setup to set the Public URL.

Configuring standard Apache mod_proxy

Using a standard Apache mod_proxy solution straight forward and allows you to configure a reverse proxy for either Tomcat or WebSphere. This provides a simple solution to configuring a proxy on any standard Apache 2.2.x http server with support for SSL and mod_proxy allowing for deployment. The underlying Apache httpd server can still be IBM HTTP Server or you can provide your own preferred Apache 2.2.x server with support for mod_ssl and mod_proxy.

Thankfully using mod_proxy is a well defined problem and you can find many useful sources of information on the internet. There are a few application and performance specific tips that we can recommend to help you avoid common problems some Jazz customers have already seen.

N.B. This is a top hit from most search engines when you search for "reverse proxy", http://www.apachetutor.org/admin/reverseproxies, feel free to follow this guidance as well but beware this is generic guidance and two major differences. First in the Jazz specific case do not enable URL rewriting as this is unnecessary since Jazz responds with the Public URL in all responses already this will only add an additional processing phase to each request impacting your performance, second do not disable compression and/or SSL unless you know what you are doing.

The configuration for a simple setup is fairly straight forward and the key pieces are to enable the modules for mod_proxy 4, mod_proxy_connect 5, and mod_proxy_http 6. These are the relevant modules for respectively performing the reverse proxy, managing the SSL tunnels through the proxy, and maintaining the HTTP connection level information associated to the connection.

The best way to understand the configuration is see a working configuration where the relevant pieces are put in place to help give a high level overview of how this works. Below are some snippets from a working httpd.conf file from an IBM HTTPServer version 7. There is very little here that is specific to IBM HTTPServer other than some minor naming and path conventions this same configuration file is equally applicable to standard Apache httpd 2.x unless otherwise noted.

Sample of the relevant excerpts using the default virtual hosts from httpd.conf:

...
ServerName clm.example.org
Listen 80
Listen 443
...
LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_http_module modules/mod_proxy_http.so
# optional: if you want to rewrite urls to public url below
LoadModule rewrite_module modules/mod_rewrite.so
...
# Optional tip: listen on 80 but forward all traffic to the
# public URL host/port regardless of how the end user arrived
<VirtualHost *:80>
  RewriteEngine on
# rewrite http => https
  RewriteCond %{SERVER_PORT} !^443$ [OR,NC]
# rewrite to public URL hostname
  RewriteCond %{HTTP_HOST}   !^clm\.example\.org [NC]
# respond with a 301 response to signify permanently
  RewriteRule ^(.*)$ https://clm.example.org$1 [R=301,L]
</VirtualHost>
...
# Prevent forwarding requests, we want reverse proxy
ProxyRequests off
# pass the Host: line from the incoming request
ProxyPreserveHost on

<Proxy *>
Order Deny,Allow
# for an internal network this Allow can be more restrictive
Allow from all
</Proxy>

# Jazz Team Server
ProxyPass /jts https://jts.example.org:9443/jts
ProxyPassReverse /jts https://jts.example.org:9443/jts

# LPA Server, here hosted on jts
ProxyPass /admin https://jts.example.org:9443/admin
ProxyPassReverse /admin https://jts.example.org:9443/admin

# Team Concert Server
ProxyPass /ccm https://ccm.example.org:9443/ccm
ProxyPassReverse /ccm https://ccm.example.org:9443/ccm

# Requirements Composer and converter server
ProxyPass /rm https://rm.example.org:9443/rm
ProxyPassReverse /rm https://rm.example.org:9443/rm
ProxyPass /converter https://rm.example.org:9443/converter
ProxyPassReverse /converter https://rm.example.org:9443/converter

# Quality Manager Server
ProxyPass /qm https://qm.example.org:9443/qm
ProxyPassReverse /qm https://qm.example.org:9443/qm

<VirtualHost *:443>
  ServerName clm.example.org
  SSLEnable
  SSLProxyEngine On
  KeyFile /opt/IBM/HTTPServer7/conf/key.kdb

# *** IHS Specific ***
# SSL CipherSpec Reference:
# http://publib.boulder.ibm.com/httpserv/ihsdiag/ihs_performance.html#SSL
  SSLCipherSpec 27
  SSLCipherSpec 21
  SSLCipherSpec 23
  SSLCipherSpec 3A
  SSLCipherSpec 34
  SSLCipherSpec 35
  
  RewriteEngine on
# rewrite to public URL hostname, this is a simple helper if you are only
# proxying for a single server, otherwise NameVirtualHost is a better 
# option for handling this
  RewriteCond %{HTTP_HOST}   !^clm\.example\.org [NC]
# respond with a 301 response to signify permanently
  RewriteRule ^(.*)$ https://clm.example.org$1 [R=301,L]
  
</VirtualHost>

Additional example using named virtual hosts:

Listen 0.0.0.0:443
## IPv6 support:
# Listen [::]:443
<VirtualHost clm01.example.com:443>
  ServerName clm01.example.com
  SSLEnable
  SSLProtocolDisable SSLv2
  KeyFile C:/Progra~1/IBM/HTTPServer7/key-clm01.kdb
  
  <IfModule mod_proxy.c>
   ProxyRequests off
   ProxyPreserveHost on

   <Proxy https://clm01.example.com/*>
      Order deny,allow
      Allow from all
   </Proxy>
   ProxyVia On

   </IfModule>
  
  # Reverse Proxy Info:
   SSLProxyEngine on
   
   ProxyPass /jts https://clm01node1.example.com:9443/jts
   ProxyPassreverse /jts https://clm01node1.example.com:9443/jts
  
   ProxyPass /ccm https://clm01node2.example.com:9443/ccm
   ProxyPassreverse /ccm https://clm01node2.example.com:9443/ccm

</VirtualHost>

<VirtualHost clm02.example.com:443>
  ServerName clm02.example.com
  SSLEnable
  SSLProtocolDisable SSLv2
  KeyFile C:/Progra~1/IBM/HTTPServer7/key-clm02.kdb
  
  <IfModule mod_proxy.c>
   ProxyRequests off
   ProxyPreserveHost on

   <Proxy https://clm02.example.com/*>
      Order deny,allow
      Allow from all
   </Proxy>
   ProxyVia On

   </IfModule>
  
  # Reverse Proxy Info:
   SSLProxyEngine on
   
   ProxyPass /jts https://clm02node1.example.com:9443/jts
   ProxyPassreverse /jts https://clm01node1.example.com:9443/jts
  
   ProxyPass /ccm https://clm02node2.example.com:9443/ccm
   ProxyPassreverse /ccm https://clm01node2.example.com:9443/ccm

</VirtualHost>

Configuring Standard Apache 2.2

The configuration is logically the same within the standard Apache 2.2 server installed with most modern operating systems or if you go to http.apache.org and download the latest version of the server as a starting point. The underlying technology is the same as IBM HTTP Server described above and each vendor generally provides their own packaged version with slight variations on the configuration file management but all the options remain the same.

Here is an example that uses the default Apache configuration shipped with CentOS 6 out of the box, to get you up and running within a few minutes by adding a new named virtual host. The default Apache install with CentOS comes configured on 80 & 443 with a self-signed certificate pre-created.

Update /etc/httpd/conf/httpd.conf

...
ServerName clm.example.org
Listen 80
Listen 443
...
LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_http_module modules/mod_proxy_http.so
# optional: if you want to rewrite urls to public url below
LoadModule rewrite_module modules/mod_rewrite.so
...
#
# Use name-based virtual hosting.
#
NameVirtualHost clm.example.org:80
NameVirtualHost clm.example.org:443

#
# Load NamedVirtualHost config files from the config directory "/etc/httpd/vhosts.d".
#
Include vhosts.d/*.conf

Add a new directory for your vhosts /etc/httpd/vhosts.d and a new configuration file /etc/httpd/vhosts.d/clm.example.org.conf. A new directory is not strictly necessary but we do need to explicitly load this file after the default /etc/httpd/conf.d/*.conf files get loaded or the server may not be fully configured yet. (In this case /etc/httpd/conf.d/ssl.conf will not have been loaded yet).

Contents of the /etc/httpd/vhosts.d/clm.example.org.conf file.

<VirtualHost clm.example.org:80>
  ServerName clm.example.org
  RewriteEngine on
## rewrite http => https
  RewriteCond %{SERVER_PORT} !^443$ [OR,NC]
## rewrite to public URL hostname
  RewriteCond %{HTTP_HOST}   !^clm\.example\.org [NC]
## respond with a 301 response to signify permanent redirect
  RewriteRule ^(.*)$ https://clm.example.org$1 [R=301,L]
</VirtualHost>

  <VirtualHost clm.example.org:443>
    ServerName clm.example.org
    ServerAdmin webmaster@clm.example.org
    DocumentRoot /var/www/vhosts/clm.example.org/htdocs
    ErrorLog logs/clm.example.org-error_log
    CustomLog logs/dummy-clm.example.org-access_log common

  <Directory /var/www/vhosts/clm.example.org/htdocs >
    Options Indexes FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
  </Directory>

  # Jazz Team Server
  ProxyPass /jts https://jts.example.org:9443/jts
  ProxyPassReverse /jts https://jts.example.org:9443/jts

  # LPA Server, here hosted on jts
  ProxyPass /admin https://jts.example.org:9443/admin
  ProxyPassReverse /admin https://jts.example.org:9443/admin

  # Team Concert Server
  ProxyPass /ccm https://ccm.example.org:9443/ccm
  ProxyPassReverse /ccm https://ccm.example.org:9443/ccm

  # Requirements Composer and converter server
  ProxyPass /rm https://rm.example.org:9443/rm
  ProxyPassReverse /rm https://rm.example.org:9443/rm
  ProxyPass /converter https://rm.example.org:9443/converter
  ProxyPassReverse /converter https://rm.example.org:9443/converter

  # Quality Manager Server
  ProxyPass /qm https://qm.example.org:9443/qm
  ProxyPassReverse /qm https://qm.example.org:9443/qm

</VirtualHost>

Additional Task Required if using Self Signed Application Server Certificates:

N.B. This is only required if you do not have a valid SSL certificate from a trusted CA installed into your Tomcat/WebSphere application server.

When a self-signed certificate is used, the proxy server needs to be able to directly connect to the underlying server with a valid SSL connection. When a trusted SSL certificate is installed, the proxy server can successfully look this up by using public Certificate Authorities to validate the certificate. In the case of a self-signed certificate, we need to take and extra step to export the current self-signed SSL certificate from our application server and import this into our proxy servers key database as a signer certificate. This allows the certificate validation to be possible without requiring a public trusted Certificate Authority.

The high-level procedure is:

1. Export the self-signed certificate from the application server in X.509 *.der format.
2. Open your Apache key database and import the certificate as Signer Certificates.
3. Repeat Steps 1 & 2 for each backend application server.
4. Save the updated key database.
5. Restart Apache.

The underlying procedure will be slightly different for Apache and OpenSSL than for IBM HTTPServer with Global Security Kit(GSK) for SSL.

IBM HTTPServer and GSK method:

The simplest method is to connect to the backend servers directly in your browser both IE and FireFox provide a simple GUI to export the certificate presented by the application server to the required X.509 *.der file format for import. Once the certificate from the target server has been exported, you need to import this as a signer certificate into your existing Apache key database. With IBM HTTPServer and access to the gui, it is possible to start ikeyman.(bat|sh), open the key database, select the signer certificate tab, and Add the corresponding *.der file(s).

This is also possible with the command line where no gui is available and you can investigate the gory details of working with the gskcmd in the WebSphere Information Center: Importing and exporting keys using the command line4.

Apache and OpenSSL Method

The analogous operations can be done with the OpenSSL tools to setup your Apache web server. With Tomcat but they must be performed via the command line. The general process is the same and your intention is to export the current Tomcat self-signed certificate and import it into the httpd keystore so that a trusted key chain can be established. There are numerous ways to do this but I have referenced a few ways that will get you there at the end of this article ( see Exporting tomcat keys to Apache httpd 8 and Securing Tomcat with Apache Web Server mod_proxy 9).

Common Errors from self-signed SSL keys

A few examples of errors that indicate that we are not correctly connecting to the back-end servers, notice the failure here are during the handshake and we can see that these are both generated by IHS where we use the GSK 7 SSL libraries.

[Fri Oct 07 14:51:10 2011] [error] SSL0266E: Handshake Failed, Could not establish SSL proxy connection.
[Fri Oct 07 14:51:10 2011] [warn] [client 127.0.0.1] [13717d0] SSL0234W: SSL Handshake Failed, The \
  certificate sent by the peer has expired or is invalid.
[Fri Oct 07 14:51:10 2011] [warn] [client 127.0.0.1] [13717d0] Certificate validation error during \
  handshake, last PKIX/RFC3280 certificate validation error was GSKVAL_ERROR_NO_CHAIN_BUILT

Another example error when the certificate is incorrect from IHS.

[Wed Nov 09 15:25:22 2011] [warn] [client 127.0.0.1] [8cf4580] [19911] Certificate validation error \
  during handshake, last PKIX/RFC3280 certificate validation error was GSKVAL_ERROR_NO_CHAIN_BUILT \
  [127.0.0.1:9443 -> 127.0.0.1:39962] [15:25:22.00026035]

Reference

Related Content

• IBM Collaboratice Lifecycle Management 3.0.1 Information Center
• jazz.net - Using content caching proxies for Jazz Source Control
• IBM WebSphere 7 Information Center
• ApacheTutor.com - Reverse Proxies
• Anuradha Ramamoorthy - How to set up a reverse proxy server for an IBM Rational Jazz Team Server
• jazz.net/wiki - Complex CALM Deploy
• IBM HTTP SSL Server Questions and Answers
• IBM WebSphere Application Server 7 Information Center - SSL HandShake Messages

Footnotes

1 jazz.net: Moving Jazz Servers and URI Stability with CLM 2011
2 Apache HTTP Server Version 2.2: configure - Configure the source tree
3 Collaborative Lifecycle Management 3.0.1 Information Center: Using a reverse proxy in your topology
4 httpd.apache.org - mod_proxy
5 httpd.apache.org - mod_proxy_connect
6 httpd.apache.org - mod_proxy_http
7 WebSphere Information Center 7: Importing and exporting keys using the command line
8 SK's Tech Blog - Exporting tomcat keys to Apache httpd
9 Web Developer - Securing Tomcat with Apache Web Server mod_proxy

Related topics: Proxy server installation, Understanding reverse proxy, Configuring Enterprise CLM Reverse Proxies: WebSphere 8 and IHS 8, Configuring Enterprise CLM Reverse Proxies: WebSphere 8.5 ND Proxy

Additional contributors: RosaNaranjo

Questions and comments:

Warning: Can't find topic Deployment.ConfigureCLMEnterpriseReverseProxyWithApacheComments