Part 3: Configuring Enterprise CLM Reverse Proxies: Apache and mod_proxy

Using a standard Apache mod_proxy solution straight forward and allows you to configure a reverse proxy for either Tomcat or WebSphere. This provides a simple solution to configuring a proxy on any standard Apache 2.2.x http server with support for SSL and mod_proxy allowing for deployment. The underlying Apache httpd server can still be IBM HTTP Server or you can provide your own preferred Apache 2.2.x server with support for mod_ssl and mod_proxy.

Thankfully using mod_proxy is a well defined problem and you can find many useful sources of information on the internet. There are a few application and performance specific tips that we can recommend to help you avoid common problems some Jazz customers have already seen.

N.B. This is a top hit from most search engines when you search for “reverse proxy”, http://www.apachetutor.org/admin/reverseproxies, feel free to follow this guidance as well but beware this is generic guidance and two major differences. First in the Jazz specific case do not enable URL rewriting as this is unnecessary since Jazz responds with the Public URL in all responses already this will only add an additional processing phase to each request impacting your performance, second do not disable compression and/or SSL unless you know what you are doing.

The configuration for a simple setup is fairly straight forward and the key pieces are to enable the modules for mod_proxy⁴, mod_proxy_connect ⁵, and mod_proxy_http ⁶. These are the relevant modules for respectively performing the reverse proxy, managing the SSL tunnels through the proxy, and maintaining the HTTP connection level information associated to the connection.

The best way to understand the configuration is see a working configuration where the relevant pieces are put in place to help give a high level overview of how this works. Below are some snippets from a working httpd.conf file from an IBM HTTPServer version 7. There is very little here that is specific to IBM HTTPServer other than some minor naming and path conventions this same configuration file is equally applicable to standard Apache httpd 2.x unless otherwise noted.

Sample of the relevant excerpts using the default virtual hosts from httpd.conf:

...  ServerName clm.example.org  Listen 80  Listen 443  ...  LoadModule ibm_ssl_module modules/mod_ibm_ssl.so  LoadModule proxy_module modules/mod_proxy.so  LoadModule proxy_connect_module modules/mod_proxy_connect.so  LoadModule proxy_http_module modules/mod_proxy_http.so  # optional: if you want to rewrite urls to public url below  LoadModule rewrite_module modules/mod_rewrite.so  ...  # Optional tip: listen on 80 but forward all traffic to the  # public URL host/port regardless of how the end user arrived  <VirtualHost *:80>    RewriteEngine on  # rewrite http => https    RewriteCond %{SERVER_PORT} !^443$ [OR,NC]  # rewrite to public URL hostname    RewriteCond %{HTTP_HOST}   !^clm.example.org [NC]  # respond with a 301 response to signify permanently    RewriteRule ^(.*)$ https://clm.example.org$1 [R=301,L]  </VirtualHost>  ...  # Prevent forwarding requests, we want reverse proxy  ProxyRequests off  # pass the Host: line from the incoming request  ProxyPreserveHost on    <Proxy *>  Order Deny,Allow  # for an internal network this Allow can be more restrictive  Allow from all  </Proxy>    # Jazz Team Server  ProxyPass /jts https://jts.example.org:9443/jts  ProxyPassReverse /jts https://jts.example.org:9443/jts    # LPA Server, here hosted on jts  ProxyPass /admin https://jts.example.org:9443/admin  ProxyPassReverse /admin https://jts.example.org:9443/admin    # Team Concert Server  ProxyPass /ccm https://ccm.example.org:9443/ccm  ProxyPassReverse /ccm https://ccm.example.org:9443/ccm    # Requirements Composer and converter server  ProxyPass /rm https://rm.example.org:9443/rm  ProxyPassReverse /rm https://rm.example.org:9443/rm  ProxyPass /converter https://rm.example.org:9443/converter  ProxyPassReverse /converter https://rm.example.org:9443/converter    # Quality Manager Server  ProxyPass /qm https://qm.example.org:9443/qm  ProxyPassReverse /qm https://qm.example.org:9443/qm    <VirtualHost *:443>    ServerName clm.example.org    SSLEnable    SSLProxyEngine On    KeyFile /opt/IBM/HTTPServer7/conf/key.kdb    # *** IHS Specific ***  # SSL CipherSpec Reference:  # http://publib.boulder.ibm.com/httpserv/ihsdiag/ihs_performance.html#SSL    SSLCipherSpec 27    SSLCipherSpec 21    SSLCipherSpec 23    SSLCipherSpec 3A    SSLCipherSpec 34    SSLCipherSpec 35        RewriteEngine on  # rewrite to public URL hostname, this is a simple helper if you are only  # proxying for a single server, otherwise NameVirtualHost is a better   # option for handling this    RewriteCond %{HTTP_HOST}   !^clm.example.org [NC]  # respond with a 301 response to signify permanently    RewriteRule ^(.*)$ https://clm.example.org$1 [R=301,L]      </VirtualHost>  

Additional example using named virtual hosts:

  Listen 0.0.0.0:443  ## IPv6 support:  # Listen [::]:443  <VirtualHost clm01.example.com:443>    ServerName clm01.example.com    SSLEnable    SSLProtocolDisable SSLv2    KeyFile C:/Progra~1/IBM/HTTPServer7/key-clm01.kdb        <IfModule mod_proxy.c>  	ProxyRequests off  	ProxyPreserveHost on    	<Proxy https://clm01.example.com/*>  		Order deny,allow  		Allow from all  	</Proxy>  	ProxyVia On    	</IfModule>        # Reverse Proxy Info:     SSLProxyEngine on          ProxyPass /jts https://clm01node1.example.com:9443/jts     ProxyPassreverse /jts https://clm01node1.example.com:9443/jts         ProxyPass /ccm https://clm01node2.example.com:9443/ccm     ProxyPassreverse /ccm https://clm01node2.example.com:9443/ccm    </VirtualHost>    <VirtualHost clm02.example.com:443>    ServerName clm02.example.com    SSLEnable    SSLProtocolDisable SSLv2    KeyFile C:/Progra~1/IBM/HTTPServer7/key-clm02.kdb        <IfModule mod_proxy.c>  	ProxyRequests off  	ProxyPreserveHost on    	<Proxy https://clm02.example.com/*>  		Order deny,allow  		Allow from all  	</Proxy>  	ProxyVia On    	</IfModule>        # Reverse Proxy Info:     SSLProxyEngine on          ProxyPass /jts https://clm02node1.example.com:9443/jts     ProxyPassreverse /jts https://clm01node1.example.com:9443/jts         ProxyPass /ccm https://clm02node2.example.com:9443/ccm     ProxyPassreverse /ccm https://clm01node2.example.com:9443/ccm    </VirtualHost>  

Configuring Standard Apache 2.2

The configuration is logically the same within the standard Apache 2.2 server installed with most modern operating systems or if you go to http.apache.org and download the latest version of the server as a starting point. The underlying technology is the same as IBM HTTP Server described above and each vendor generally provides their own packaged version with slight variations on the configuration file management but all the options remain the same.

Here is an example that uses the default Apache configuration shipped with CentOS 6 out of the box, to get you up and running within a few minutes by adding a new named virtual host. The default Apache install with CentOS comes configured on 80 & 443 with a self-signed certificate pre-created.

Update /etc/httpd/conf/httpd.conf

  ...  ServerName clm.example.org  Listen 80  Listen 443  ...  LoadModule ibm_ssl_module modules/mod_ibm_ssl.so  LoadModule proxy_module modules/mod_proxy.so  LoadModule proxy_connect_module modules/mod_proxy_connect.so  LoadModule proxy_http_module modules/mod_proxy_http.so  # optional: if you want to rewrite urls to public url below  LoadModule rewrite_module modules/mod_rewrite.so  ...  #  # Use name-based virtual hosting.  #  NameVirtualHost clm.example.org:80  NameVirtualHost clm.example.org:443    #  # Load NamedVirtualHost config files from the config directory "/etc/httpd/vhosts.d".  #  Include vhosts.d/*.conf  

Add a new directory for your vhosts /etc/httpd/vhosts.d and a new configuration file /etc/httpd/vhosts.d/clm.example.org.conf. A new directory is not strictly necessary but we do need to explicitly load this file after the default /etc/httpd/conf.d/*.conf files get loaded or the server may not be fully configured yet. (In this case /etc/httpd/conf.d/ssl.conf will not have been loaded yet).

Contents of the /etc/httpd/vhosts.d/clm.example.org.conf file.

  <VirtualHost clm.example.org:80>    ServerName clm.example.org    RewriteEngine on  ## rewrite http => https    RewriteCond %{SERVER_PORT} !^443$ [OR,NC]  ## rewrite to public URL hostname    RewriteCond %{HTTP_HOST}   !^clm.example.org [NC]  ## respond with a 301 response to signify permanent redirect    RewriteRule ^(.*)$ https://clm.example.org$1 [R=301,L]  </VirtualHost>      <VirtualHost clm.example.org:443>      ServerName clm.example.org      ServerAdmin webmaster@clm.example.org      DocumentRoot /var/www/vhosts/clm.example.org/htdocs      ErrorLog logs/clm.example.org-error_log      CustomLog logs/dummy-clm.example.org-access_log common      <Directory /var/www/vhosts/clm.example.org/htdocs >      Options Indexes FollowSymLinks      AllowOverride None      Order allow,deny      Allow from all    </Directory>      # Jazz Team Server    ProxyPass /jts https://jts.example.org:9443/jts    ProxyPassReverse /jts https://jts.example.org:9443/jts      # LPA Server, here hosted on jts    ProxyPass /admin https://jts.example.org:9443/admin    ProxyPassReverse /admin https://jts.example.org:9443/admin      # Team Concert Server    ProxyPass /ccm https://ccm.example.org:9443/ccm    ProxyPassReverse /ccm https://ccm.example.org:9443/ccm      # Requirements Composer and converter server    ProxyPass /rm https://rm.example.org:9443/rm    ProxyPassReverse /rm https://rm.example.org:9443/rm    ProxyPass /converter https://rm.example.org:9443/converter    ProxyPassReverse /converter https://rm.example.org:9443/converter      # Quality Manager Server    ProxyPass /qm https://qm.example.org:9443/qm    ProxyPassReverse /qm https://qm.example.org:9443/qm    </VirtualHost>  

Additional Task Required if using Self Signed Application Server Certificates:

N.B. This is only required if you do not have a valid SSL certificate from a trusted CA installed into your Tomcat/WebSphere application server.

When a self-signed certificate is used, the proxy server needs to be able to directly connect to the underlying server with a valid SSL connection. When a trusted SSL certificate is installed, the proxy server can successfully look this up by using public Certificate Authorities to validate the certificate. In the case of a self-signed certificate, we need to take and extra step to export the current self-signed SSL certificate from our application server and import this into our proxy servers key database as a signer certificate. This allows the certificate validation to be possible without requiring a public trusted Certificate Authority.

The high-level procedure is:

1. Export the self-signed certificate from the application server in X.509 *.der format.
2. Open your Apache key database and import the certificate as Signer Certificates.
3. Repeat Steps 1 & 2 for each backend application server.
4. Save the updated key database.
5. Restart Apache.

The underlying procedure will be slightly different for Apache and OpenSSL than for IBM HTTPServer with Global Security Kit(GSK) for SSL.

IBM HTTPServer and GSK method:

The simplest method is to connect to the backend servers directly in your browser both IE and FireFox provide a simple GUI to export the certificate presented by the application server to the required X.509 *.der file format for import. Once the certificate from the target server has been exported, you need to import this as a signer certificate into your existing Apache key database. With IBM HTTPServer and access to the gui, it is possible to start ikeyman.(bat|sh), open the key database, select the signer certificate tab, and Add the corresponding *.der file(s).

This is also possible with the command line where no gui is available and you can investigate the gory details of working with the gskcmd in the WebSphere Information Center: Importing and exporting keys using the command line⁴.

Apache and OpenSSL Method

The analogous operations can be done with the OpenSSL tools to setup your Apache web server. With Tomcat but they must be performed via the command line. The general process is the same and your intention is to export the current Tomcat self-signed certificate and import it into the httpd keystore so that a trusted key chain can be established. There are numerous ways to do this but I have referenced a few ways that will get you there at the end of this article ( see Exporting tomcat keys to Apache httpd ⁸ and Securing Tomcat with Apache Web Server mod_proxy ⁹).

---

Common Errors from self-signed SSL keys

A few examples of errors that indicate that we are not correctly connecting to the back-end servers, notice the failure here are during the handshake and we can see that these are both generated by IHS where we use the GSK 7 SSL libraries.

  [Fri Oct 07 14:51:10 2011] [error] SSL0266E: Handshake Failed, Could not establish SSL proxy connection.  [Fri Oct 07 14:51:10 2011] [warn] [client 127.0.0.1] [13717d0] SSL0234W: SSL Handshake Failed, The     certificate sent by the peer has expired or is invalid.  [Fri Oct 07 14:51:10 2011] [warn] [client 127.0.0.1] [13717d0] Certificate validation error during     handshake, last PKIX/RFC3280 certificate validation error was GSKVAL_ERROR_NO_CHAIN_BUILT  

Another example error when the certificate is incorrect from IHS.

  [Wed Nov 09 15:25:22 2011] [warn] [client 127.0.0.1] [8cf4580] [19911] Certificate validation error     during handshake, last PKIX/RFC3280 certificate validation error was GSKVAL_ERROR_NO_CHAIN_BUILT     [127.0.0.1:9443 -> 127.0.0.1:39962] [15:25:22.000260355]