![](/_scripts/dojo_dist/jazz/cs/graphics/Library/bttn_article52.gif)
Part 3: Configuring Enterprise CLM Reverse Proxies: Apache and mod_proxy
Sean Wilbur, IBM Rational
Last updated: August 28, 2012
Build basis: Collaborative Lifecycle Management Platform 3.0.1
![](https://jazz.net/downloads/images/icon-warn.gif)
All deployment guidance and best practice will be published in the Deployment wiki rather than as Jazz.net articles going forward.
This article outlines the process for configuring a reverse proxy to support a consistent Public URL1 in a flexible deployment topology. Provided here are a few sample deployment options to reference when configuring your own reverse proxy configuration.
Other articles in this series:
- Part 1: Configuring Enterprise CLM Reverse Proxies: Understanding Reverse Proxy
- Part 2: WebSphere and IHS Plugin method
Table of Contents
- Prerequisites
- Configuring standard Apache mod_proxy
- Common Errors from self-signed SSL keys
- Reference
Prerequisites
- Apache httpd basic administrative skills are not covered here.
- Write access Apache/IHS machine configuration directory:
We will need to have write access to the location of httpd.conf(apache/conf) and the key.kdb/key.sth(few common options apache, apache/conf, /etc/ssl ) files are located.
- Apache httpd server with mod_ssl & mod_proxy support:
Next we need to ensure that the version of apache we have can support the modules we need. While both of these options are common it still requires the builder to select them as neither
--enable-proxy
or--enable-ssl
are default compile options 2. This is generally the case is most packaged versions of Apache today but something to check if you don’t see the modules available in yourmodules
directory. - Configure this external facing httpd server to support SSL:
Configuring the external facing Apache server for SSL is required and well documented for both Apache and IHS.
- Running Jazz deployment (configured or not):
- An existing configured environment that has already been setup with a defined Public URL and your configuring the proxy to allow for a distributed deployment3.
- A new install that will required the reverse proxy before you can complete
/jts/setup
to set the Public URL.
Configuring standard Apache mod_proxy
Using a standard Apache mod_proxy solution straight forward and allows you to configure a reverse proxy for either Tomcat or WebSphere. This provides a simple solution to configuring a proxy on any standard Apache 2.2.x http server with support for SSL and mod_proxy allowing for deployment. The underlying Apache httpd server can still be IBM HTTP Server or you can provide your own preferred Apache 2.2.x server with support for mod_ssl and mod_proxy.
Thankfully using mod_proxy
is a well defined problem and you can find many useful sources of information on the internet. There are a few application and performance specific tips that we can recommend to help you avoid common problems some Jazz customers have already seen.
N.B. This is a top hit from most search engines when you search for “reverse proxy”, http://www.apachetutor.org/admin/reverseproxies, feel free to follow this guidance as well but beware this is generic guidance and two major differences. First in the Jazz specific case do not enable URL rewriting as this is unnecessary since Jazz responds with the Public URL in all responses already this will only add an additional processing phase to each request impacting your performance, second do not disable compression and/or SSL unless you know what you are doing.
The configuration for a simple setup is fairly straight forward and the key pieces are to enable the modules for mod_proxy
4, mod_proxy_connect
5, and mod_proxy_http
6. These are the relevant modules for respectively performing the reverse proxy, managing the SSL tunnels through the proxy, and maintaining the HTTP connection level information associated to the connection.
The best way to understand the configuration is see a working configuration where the relevant pieces are put in place to help give a high level overview of how this works. Below are some snippets from a working httpd.conf file from an IBM HTTPServer version 7. There is very little here that is specific to IBM HTTPServer other than some minor naming and path conventions this same configuration file is equally applicable to standard Apache httpd 2.x unless otherwise noted.
Sample of the relevant excerpts using the default virtual hosts from httpd.conf:
... ServerName clm.example.org Listen 80 Listen 443 ... LoadModule ibm_ssl_module modules/mod_ibm_ssl.so LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_connect_module modules/mod_proxy_connect.so LoadModule proxy_http_module modules/mod_proxy_http.so # optional: if you want to rewrite urls to public url below LoadModule rewrite_module modules/mod_rewrite.so ... # Optional tip: listen on 80 but forward all traffic to the # public URL host/port regardless of how the end user arrived <VirtualHost *:80> RewriteEngine on # rewrite http => https RewriteCond %{SERVER_PORT} !^443$ [OR,NC] # rewrite to public URL hostname RewriteCond %{HTTP_HOST} !^clm.example.org [NC] # respond with a 301 response to signify permanently RewriteRule ^(.*)$ https://clm.example.org$1 [R=301,L] </VirtualHost> ... # Prevent forwarding requests, we want reverse proxy ProxyRequests off # pass the Host: line from the incoming request ProxyPreserveHost on <Proxy *> Order Deny,Allow # for an internal network this Allow can be more restrictive Allow from all </Proxy> # Jazz Team Server ProxyPass /jts https://jts.example.org:9443/jts ProxyPassReverse /jts https://jts.example.org:9443/jts # LPA Server, here hosted on jts ProxyPass /admin https://jts.example.org:9443/admin ProxyPassReverse /admin https://jts.example.org:9443/admin # Team Concert Server ProxyPass /ccm https://ccm.example.org:9443/ccm ProxyPassReverse /ccm https://ccm.example.org:9443/ccm # Requirements Composer and converter server ProxyPass /rm https://rm.example.org:9443/rm ProxyPassReverse /rm https://rm.example.org:9443/rm ProxyPass /converter https://rm.example.org:9443/converter ProxyPassReverse /converter https://rm.example.org:9443/converter # Quality Manager Server ProxyPass /qm https://qm.example.org:9443/qm ProxyPassReverse /qm https://qm.example.org:9443/qm <VirtualHost *:443> ServerName clm.example.org SSLEnable SSLProxyEngine On KeyFile /opt/IBM/HTTPServer7/conf/key.kdb # *** IHS Specific *** # SSL CipherSpec Reference: # http://publib.boulder.ibm.com/httpserv/ihsdiag/ihs_performance.html#SSL SSLCipherSpec 27 SSLCipherSpec 21 SSLCipherSpec 23 SSLCipherSpec 3A SSLCipherSpec 34 SSLCipherSpec 35 RewriteEngine on # rewrite to public URL hostname, this is a simple helper if you are only # proxying for a single server, otherwise NameVirtualHost is a better # option for handling this RewriteCond %{HTTP_HOST} !^clm.example.org [NC] # respond with a 301 response to signify permanently RewriteRule ^(.*)$ https://clm.example.org$1 [R=301,L] </VirtualHost>
Additional example using named virtual hosts:
Listen 0.0.0.0:443 ## IPv6 support: # Listen [::]:443 <VirtualHost clm01.example.com:443> ServerName clm01.example.com SSLEnable SSLProtocolDisable SSLv2 KeyFile C:/Progra~1/IBM/HTTPServer7/key-clm01.kdb <IfModule mod_proxy.c> ProxyRequests off ProxyPreserveHost on <Proxy https://clm01.example.com/*> Order deny,allow Allow from all </Proxy> ProxyVia On </IfModule> # Reverse Proxy Info: SSLProxyEngine on ProxyPass /jts https://clm01node1.example.com:9443/jts ProxyPassreverse /jts https://clm01node1.example.com:9443/jts ProxyPass /ccm https://clm01node2.example.com:9443/ccm ProxyPassreverse /ccm https://clm01node2.example.com:9443/ccm </VirtualHost> <VirtualHost clm02.example.com:443> ServerName clm02.example.com SSLEnable SSLProtocolDisable SSLv2 KeyFile C:/Progra~1/IBM/HTTPServer7/key-clm02.kdb <IfModule mod_proxy.c> ProxyRequests off ProxyPreserveHost on <Proxy https://clm02.example.com/*> Order deny,allow Allow from all </Proxy> ProxyVia On </IfModule> # Reverse Proxy Info: SSLProxyEngine on ProxyPass /jts https://clm02node1.example.com:9443/jts ProxyPassreverse /jts https://clm01node1.example.com:9443/jts ProxyPass /ccm https://clm02node2.example.com:9443/ccm ProxyPassreverse /ccm https://clm01node2.example.com:9443/ccm </VirtualHost>
Configuring Standard Apache 2.2
The configuration is logically the same within the standard Apache 2.2 server installed with most modern operating systems or if you go to http.apache.org and download the latest version of the server as a starting point. The underlying technology is the same as IBM HTTP Server described above and each vendor generally provides their own packaged version with slight variations on the configuration file management but all the options remain the same.
Here is an example that uses the default Apache configuration shipped with CentOS 6 out of the box, to get you up and running within a few minutes by adding a new named virtual host. The default Apache install with CentOS comes configured on 80 & 443 with a self-signed certificate pre-created.
Update /etc/httpd/conf/httpd.conf
... ServerName clm.example.org Listen 80 Listen 443 ... LoadModule ibm_ssl_module modules/mod_ibm_ssl.so LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_connect_module modules/mod_proxy_connect.so LoadModule proxy_http_module modules/mod_proxy_http.so # optional: if you want to rewrite urls to public url below LoadModule rewrite_module modules/mod_rewrite.so ... # # Use name-based virtual hosting. # NameVirtualHost clm.example.org:80 NameVirtualHost clm.example.org:443 # # Load NamedVirtualHost config files from the config directory "/etc/httpd/vhosts.d". # Include vhosts.d/*.conf
Add a new directory for your vhosts /etc/httpd/vhosts.d
and a new configuration file /etc/httpd/vhosts.d/clm.example.org.conf
. A new directory is not strictly necessary but we do need to explicitly load this file after the default /etc/httpd/conf.d/*.conf
files get loaded or the server may not be fully configured yet. (In this case /etc/httpd/conf.d/ssl.conf
will not have been loaded yet).
Contents of the /etc/httpd/vhosts.d/clm.example.org.conf
file.
<VirtualHost clm.example.org:80> ServerName clm.example.org RewriteEngine on ## rewrite http => https RewriteCond %{SERVER_PORT} !^443$ [OR,NC] ## rewrite to public URL hostname RewriteCond %{HTTP_HOST} !^clm.example.org [NC] ## respond with a 301 response to signify permanent redirect RewriteRule ^(.*)$ https://clm.example.org$1 [R=301,L] </VirtualHost> <VirtualHost clm.example.org:443> ServerName clm.example.org ServerAdmin webmaster@clm.example.org DocumentRoot /var/www/vhosts/clm.example.org/htdocs ErrorLog logs/clm.example.org-error_log CustomLog logs/dummy-clm.example.org-access_log common <Directory /var/www/vhosts/clm.example.org/htdocs > Options Indexes FollowSymLinks AllowOverride None Order allow,deny Allow from all </Directory> # Jazz Team Server ProxyPass /jts https://jts.example.org:9443/jts ProxyPassReverse /jts https://jts.example.org:9443/jts # LPA Server, here hosted on jts ProxyPass /admin https://jts.example.org:9443/admin ProxyPassReverse /admin https://jts.example.org:9443/admin # Team Concert Server ProxyPass /ccm https://ccm.example.org:9443/ccm ProxyPassReverse /ccm https://ccm.example.org:9443/ccm # Requirements Composer and converter server ProxyPass /rm https://rm.example.org:9443/rm ProxyPassReverse /rm https://rm.example.org:9443/rm ProxyPass /converter https://rm.example.org:9443/converter ProxyPassReverse /converter https://rm.example.org:9443/converter # Quality Manager Server ProxyPass /qm https://qm.example.org:9443/qm ProxyPassReverse /qm https://qm.example.org:9443/qm </VirtualHost>
Additional Task Required if using Self Signed Application Server Certificates:
N.B. This is only required if you do not have a valid SSL certificate from a trusted CA installed into your Tomcat/WebSphere application server.
When a self-signed certificate is used, the proxy server needs to be able to directly connect to the underlying server with a valid SSL connection. When a trusted SSL certificate is installed, the proxy server can successfully look this up by using public Certificate Authorities to validate the certificate. In the case of a self-signed certificate, we need to take and extra step to export the current self-signed SSL certificate from our application server and import this into our proxy servers key database as a signer certificate. This allows the certificate validation to be possible without requiring a public trusted Certificate Authority.
The high-level procedure is:
- Export the self-signed certificate from the application server in X.509 *.der format.
- Open your Apache key database and import the certificate as Signer Certificates.
- Repeat Steps 1 & 2 for each backend application server.
- Save the updated key database.
- Restart Apache.
The underlying procedure will be slightly different for Apache and OpenSSL than for IBM HTTPServer with Global Security Kit(GSK) for SSL.
IBM HTTPServer and GSK method:
The simplest method is to connect to the backend servers directly in your browser both IE and FireFox provide a simple GUI to export the certificate presented by the application server to the required X.509 *.der file format for import. Once the certificate from the target server has been exported, you need to import this as a signer certificate into your existing Apache key database. With IBM HTTPServer and access to the gui, it is possible to start ikeyman.(bat|sh), open the key database, select the signer certificate tab, and Add the corresponding *.der file(s).
This is also possible with the command line where no gui is available and you can investigate the gory details of working with the gskcmd in the WebSphere Information Center: Importing and exporting keys using the command line4.
Apache and OpenSSL Method
The analogous operations can be done with the OpenSSL tools to setup your Apache web server. With Tomcat but they must be performed via the command line. The general process is the same and your intention is to export the current Tomcat self-signed certificate and import it into the httpd keystore so that a trusted key chain can be established. There are numerous ways to do this but I have referenced a few ways that will get you there at the end of this article ( see Exporting tomcat keys to Apache httpd 8 and Securing Tomcat with Apache Web Server mod_proxy 9).
Common Errors from self-signed SSL keys
A few examples of errors that indicate that we are not correctly connecting to the back-end servers, notice the failure here are during the handshake and we can see that these are both generated by IHS where we use the GSK 7 SSL libraries.
[Fri Oct 07 14:51:10 2011] [error] SSL0266E: Handshake Failed, Could not establish SSL proxy connection. [Fri Oct 07 14:51:10 2011] [warn] [client 127.0.0.1] [13717d0] SSL0234W: SSL Handshake Failed, The certificate sent by the peer has expired or is invalid. [Fri Oct 07 14:51:10 2011] [warn] [client 127.0.0.1] [13717d0] Certificate validation error during handshake, last PKIX/RFC3280 certificate validation error was GSKVAL_ERROR_NO_CHAIN_BUILT
Another example error when the certificate is incorrect from IHS.
[Wed Nov 09 15:25:22 2011] [warn] [client 127.0.0.1] [8cf4580] [19911] Certificate validation error during handshake, last PKIX/RFC3280 certificate validation error was GSKVAL_ERROR_NO_CHAIN_BUILT [127.0.0.1:9443 -> 127.0.0.1:39962] [15:25:22.000260355]
Reference
Related Content
- IBM Collaboratice Lifecycle Management 3.0.1 Information Center
- jazz.net – Using content caching proxies for Jazz Source Control
- IBM WebSphere 7 Information Center
- ApacheTutor.com – Reverse Proxies
- Anuradha Ramamoorthy – How to set up a reverse proxy server for an IBM Rational Jazz Team Server
- jazz.net/wiki – Complex CALM Deploy
- IBM HTTP SSL Server Questions and Answers
- IBM WebSphere Application Server 7 Information Center – SSL HandShake Messages
Footnotes
- jazz.net: Moving Jazz Servers and URI Stability with CLM 2011
- Apache HTTP Server Version 2.2: configure – Configure the source tree
- Collaborative Lifecycle Management 3.0.1 Information Center: Using a reverse proxy in your topology
- httpd.apache.org – mod_proxy
- httpd.apache.org – mod_proxy_connect
- httpd.apache.org – mod_proxy_http
- WebSphere Information Center 7: Importing and exporting keys using the command line
- SK’s Tech Blog – Exporting tomcat keys to Apache httpd
- Web Developer – Securing Tomcat with Apache Web Server mod_proxy
About the author
Sean Wilbur works in the Rational Unleash the Labs team, the Jazz JumpStart, and Rational Services teams providing support for clients adopting and evaluating Rational solutions across the globe.
Copyright © 2012 IBM Rational