Configuring Rate Limiting for ELM Applications

Authors: ArtaChaudhury, BharathRao , RalphSchoon
Build basis: Engineering Lifecycle Management 7.0.3 and higher

Rate limiting is a technique used to control the rate of incoming or outgoing traffic to or from a system by imposing restrictions on the number of requests within a specified time frame. Its primary purpose is to prevent resource exhaustion, protect against abuse, and ensure fair usage among users or clients.

ELM Applications, underlying Liberty Application Server and the supported Reverse Proxy (IBM HTTP Server) does not include / support Rate limiting. There are third party and open-source software like HAProxy

We have performed a simple configuration of HAProxy with ELM applications and documented instructions of the setup in this article

Introduction and Scope of Support

For open-source software, including HAProxy, the following IBM Policy applies: IBM Open Source and Third-party software policy

We have performed a simple rate limiting configuration using HAProxy with ELM applications and documented the instructions of the setup and use cases in this article. For detailed instructions please visit http://www.haproxy.org/

HAProxy is a free and open source software that provides a high availability load balancer and reverse proxy. It supports a rich set of Load Balancing algorithms and the default is Leastconn. Since we have tested the use of HAProxy with EWM/ETM Clustering and Load Balancing LQE Application, the same is being used for rate limiting. HAProxy is not supported on Microsoft Windows Operating System. You can continue to the next step if your environment is Linux based.

Install and Setup HAProxy

The steps provided this section is a simple setup of HAProxy. For detailed instructions please visit http://www.haproxy.org/. The Idea of this setup is to introduce a HAProxy layer in between IBM HTTP Server and the IBM Liberty Server hosting ELM Application.

Install HAProxy

You need a Linux based server in your environment to install and configure HAProxy. Run the following commands

   # yum update
   # yum install haproxy    

Create Open SSL Certificates for HAProxy

Generate SSL Certificates to be used with HAProxy via OpenSSL

   # mkdir /etc/haproxy/ssl
   # cd /etc/haproxy/ssl
   # openssl req -newkey rsa:3072 -sha256 -new -x509 -days 3652 -nodes -out haproxy.crt -keyout haproxy.key
   # cat haproxy.crt haproxy.key > haproxy.pem
   # chmod +rx haproxy.*   

Import this certificate and key file into IBM HTTP Server certificate kdb file and the Plugin kdb file.

Edit/Create HAProxy config file

Here is a sample haproxy.cfg file for load balancing 2 LQE nodes. You could change the ports (8080, 8443, 1936) to the ports of your choice and the user/group as well. In addition, change the path to the SSL certificate to the one created in the previous step.

• # vi /etc/haproxy/haproxy.cfg

global
    
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
# utilize system-wide crypto-policies
    ssl-default-bind-ciphers PROFILE=SYSTEM
    ssl-default-server-ciphers PROFILE=SYSTEM
    lua-load /etc/haproxy/delay.lua

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
    mode    http
    stats   enable
    stats   uri /stats
    stats   realm Haproxy\ Statistics
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
#frontend  lanneluc-proxy 
#   bind *:9442 ssl crt /etc/haproxy/ssl/proxy.pem no-sslv3 alpn h2,http/1.1
#   log      global
#   option   httplog
#   mode     http
#   capture  cookie SERVERID len 32
#   redirect scheme https if !{ ssl_fc }
#   maxconn 2000
#   stick-table type ip size 100k expire 600s store http_req_rate(120s)
#   http-request track-sc0 src
#   http-request deny deny_status 429 if {  sc_http_req_rate(0) gt 1000 }
#   default_backend ccm


frontend stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST
    stats auth root:test123

frontend  lanneluc-proxy
   bind *:443 ssl crt /etc/haproxy/ssl/haproxy.pem no-sslv3 alpn h2,http/1.1
   log      global
   option   httplog
   mode     http
   capture  cookie SERVERID len 32
   redirect scheme https if !{ ssl_fc }
   maxconn 2000
   #TODO add a redirect for paths ending in ccm or jts
   use_backend ccm if { path_beg /ccm/ }
   
backend ccm
   option forwardfor
   http-request set-header X-Forwarded-Port %[dst_port]
   http-request set-header X-Forwarded-Port %[dst_port]
   http-request add-header X-Forwarded-Proto https if { ssl_fc }
   fullconn 1000
   balance leastconn
   cookie SERVERID insert indirect nocache
   timeout queue 300s
   server ccm c69878v1.fyre.ibm.com:9443 minconn 20  maxconn 100 ssl check cookie ccm1 verify none

listen statistics
   bind *:1936 
   stats uri /
   stats admin  if TRUE
   stats enable
   stats hide-version
   stats auth root:clu8ter8
   stats refresh 

Enable and start HAProxy server

Run the following commands to start the HAProxy Server, enable it to auto start during machine startup and to check status of the HAProxy server

   # systemctl start haproxy
   # systemctl enable haproxy
   # systemctl status haproxy 

Update IBM HTTP Server

Edit the merged plugin file for IBM HTTP Server and update the host for EWM host to the HAProxy setup.

Enable Rate Limiting for ELM Applications

We have documented sample configurations to use rate limiting with HAProxy for CCM Application:

Use Case 1: Limit number of request

We have the capability to restrict the number of requests for any given service, and if the threshold is exceeded, we can implement specific actions. For instance, if there is a defined limit of 5 requests for the CCM dashboard service, should the server receive more than 5 requests for this service, we can respond with a 429 error indicating 'Too Many Requests,' effectively managing and controlling the traffic to ensure optimal service performance.

a) Create a mapping file called rates.map inside the folder =/etc/haproxy/=and include the following lines:

dashboard 4
Dashboard 4

This will allow 4 requests to any HTTP calls that include dashboard

b) We will now configure HAProxy to use the file and set to configuration for the use case. Here is the excerpt of the changes to be made to the default configuration provided above:

frontend  lanneluc-proxy
   bind *:443 ssl crt /etc/haproxy/ssl/haproxy.pem no-sslv3 alpn h2,http/1.1
   log      global
   option   httplog
   mode     http
   capture  cookie SERVERID len 32
   redirect scheme https if !{ ssl_fc }
   maxconn 2000
# key is a binary value, value is the http_req_rate (in built function)
   stick-table type binary len 8 size 100k expire 15s store http_req_rate(180s)
   

   # track by base32+src. base32 is a shorthand for host header and URL path
   # To get a URL Path, remove the scheme:host+port part of the url. The rest of 
   # the string is the path.
   http-request track-sc0 base32+src
  
   # Get rate limit for path by checking rates.map file. In this file, we have stored 
   # parts of the path like com.ibm.team.dashboard.service or com.ibm.team.filesystem.service
   # We do a pattern match on the path with the entries in the map (map_sub_int).
   # You can also have full paths specified in the rates.map file and retreive values based  
   # on a full match
   http-request set-var(req.rate_limit) path,map_sub_int(/etc/haproxy/rates.map,2000)

   # Get the value for client's current request rate
   http-request set-var(req.request_rate) base32+src,table_http_req_rate()

   # create a acl by subtracting current request rate from limit 
   acl rate_abuse_acl var(req.rate_limit),sub(req.request_rate) lt 0
 
   # Deny if the limit is exceeded
   # You can also slow down the request instead of denying the limit
   http-request deny deny_status 429 if rate_abuse_acl
   #TODO add a redirect for paths ending in ccm or jts
   use_backend ccm if { path_beg /ccm/ }
   use_backend jts if { path_beg /jts/ }
   #use_backend jts if { path_beg /gc/ }
   #Custom error page if the paths don't match either of the apps
  # errorfile 503 /etc/haproxy/errorfiles/503.http

Heading 1

External links:

• IBM

Additional contributors: TWikiUser, TWikiUser