Just recently Nginx rate limit by user agent (control bots) which is all cool and handy, but what if you have a number of Nginx behind HAProxy and want to offload some the job to it? Fortunately HAProxy is very easy on configuration and very flexible on ACLs. Here is a simple example on how to do different blacklists and rate limiting (just a part of configuration, apply where appropriate):
frontend http *:80
description Incoming traffic to port 80
# IP address white/blacklist
tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
tcp-request connection reject if { src -f /etc/haproxy/blacklist.lst }
# Max possible time delay on inspection
tcp-request inspect-delay 15s
# ACLs for blacklist UAs and Paths
acl abuse_ua hdr_sub(user-agent) -f /etc/haproxy/blacklist_ua.lst
acl abuse_path path_beg -f /etc/haproxy/blacklist_path.lst
# Reject blacklisted UAs and Paths
tcp-request content reject if abuse_ua
tcp-request content reject if abuse_path
# At most 10 concurrent connections from a client
acl too_fast fe_sess_rate ge 10
# Effectively working as a delay mechanism for clients that are too fast
tcp-request inspect-delay 1000ms
# Fast-path - accept connection if it's not this troublesome client
tcp-request content accept unless too_fast
# The very fast client gets here meaning they have to wait full inspect-delay
tcp-request content accept if WAIT_END
Whenever you refer to a list file to search for a value from your configuration, make sure the file is actually in place (even if it’s empty), otherwise you will fail.
The only limitation for the above is that you can’t really check headers if you are using HAProxy SSL frontend with SSL SNI, by in that case you can still implement the limits on Nginx side. The fe_sess_rate limit though is still applicable.
One note that I forgot to mention in my previous post on Nginx rate limits, you can also adjust it to work based on requested paths, not only user agents.
P.S.: When dealing with configuration changes, make sure to check the validity of the config file after changes before restarting/reloading the service. You can do it with haproxy -f /etc/haproxy/haproxy.cfg for HAProxy or nginx -t for Nginx.