Edge rate limiting and bot filtering for APIs and login pages

TL;DR — Put rate limiting and bot filtering at the first HTTP hop that terminates client traffic: your CDN/WAF if you have one, otherwise nginx. The most common failure is trusting the wrong client IP header, which collapses all users behind the proxy into one bucket and causes mass 429s. Reading time: ~5 min

Goal

When you finish, your API and login page will reject abusive bursts at the edge with 429 Too Many Requests, challenge or block obvious bots before they reach the app, and preserve real client IPs so legitimate users are not all rate-limited as one source.

Prerequisites

• Access to the edge that receives public traffic: your CDN/WAF dashboard or the nginx host with sudo/root
• nginx >= 1.20 if you are enforcing at nginx — check with:

nginx -v

• A way to reload nginx without interrupting traffic:

sudo systemctl status nginx

• Your app domains and paths, at minimum: API prefix (example: /api/) and login path (example: /login or /oauth/token)
• The proxy IP ranges for any upstream CDN/load balancer you trust; download them from your provider docs before editing set_real_ip_from
• A test client with curl >= 8 — check with:

curl --version

• Access to edge logs or nginx access logs:

sudo tail -f /var/log/nginx/access.log

Steps

Step 1: Confirm whether nginx sees the real client IP

Run these commands from a client outside your server network:

curl -sS -o /dev/null https://api.example.com/health
sudo tail -n 5 /var/log/nginx/access.log

If your log format includes $remote_addr, you should see the real public client IP, not a private address or one repeated for every request.

If you need a temporary log format that shows both direct and forwarded IPs, add this and reload:

log_format edge '$remote_addr realip=$realip_remote_addr xff="$http_x_forwarded_for" '
                '[$time_local] "$request" $status $body_bytes_sent';
access_log /var/log/nginx/access.log edge;

sudo nginx -t && sudo systemctl reload nginx

Success looks like syntax is ok and test is successful.

Step 2: Trust only your proxy and restore the client IP

Edit your nginx http {} block, usually in /etc/nginx/nginx.conf:

http {
    set_real_ip_from 203.0.113.0/24;
    set_real_ip_from 198.51.100.0/24;
    real_ip_header X-Forwarded-For;
    real_ip_recursive on;
}

Replace the CIDRs with your actual proxy/CDN egress ranges only.

Then test and reload:

sudo nginx -t
sudo systemctl reload nginx

Success looks like:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Step 3: Add separate rate limits for API and login traffic

Create /etc/nginx/conf.d/edge-rate-limit.conf:

limit_req_zone $binary_remote_addr zone=api_per_ip:20m rate=10r/s;
limit_req_zone $binary_remote_addr zone=login_per_ip:20m rate=5r/m;
limit_req_status 429;

map $http_user_agent $bad_ua {
    default 0;
    ~*(?:curl|wget|python-requests|scrapy|sqlmap|nikto|masscan|nmap) 1;
}

Then add this inside the relevant server {} block:

location /api/ {
    limit_req zone=api_per_ip burst=20 nodelay;
    proxy_pass http://app_upstream;
}

location = /login {
    limit_req zone=login_per_ip burst=10 nodelay;
    if ($bad_ua) { return 403; }
    proxy_pass http://app_upstream;
}

location = /oauth/token {
    limit_req zone=login_per_ip burst=10 nodelay;
    if ($bad_ua) { return 403; }
    proxy_pass http://app_upstream;
}

Test and reload:

sudo nginx -t && sudo systemctl reload nginx

Success means nginx reloads cleanly and new requests start returning 429 when limits are exceeded.

⚠️ A bad nginx reload can take your edge offline. Do not skip nginx -t; fix syntax errors before reloading.

Step 4: Put equivalent rules in your CDN/WAF if you have one

In your provider dashboard, create two rules at the edge before origin routing. The exact menu varies by provider; use the rate limiting/security rules area (for example, in many dashboards: Security or WAF → Rate limiting / Custom rules).

Create these literal rules:

Success looks like both rules are enabled and ordered before generic allow rules.

Step 5: Exempt health checks and trusted internal callers

If your load balancer or uptime monitor hits /health, do not burn rate-limit budget on it. Add explicit allow locations before limited ones:

location = /health {
    access_log off;
    proxy_pass http://app_upstream;
}

For internal callers behind fixed egress IPs, create a whitelist map:

geo $trusted_client {
    default 0;
    192.0.2.10/32 1;
    192.0.2.11/32 1;
}

map $trusted_client $limit_key {
    1 "";
    0 $binary_remote_addr;
}

limit_req_zone $limit_key zone=api_per_ip:20m rate=10r/s;

Reload nginx:

sudo nginx -t && sudo systemctl reload nginx

Success means health checks stay 200 and trusted IPs are not throttled.

Step 6: Test the limits and bot filter from a client

Run a burst against the API:

seq 1 40 | xargs -I{} -P20 curl -s -o /dev/null -w "%{http_code}\n" https://api.example.com/api/ping | sort | uniq -c

Expected success shape:

     22 200
     18 429

Test the login bot filter with a bad user agent:

curl -i -A "python-requests/2.32" https://app.example.com/login

Expected output shape:

HTTP/2 403
content-type: text/html
content-length: 153

Test a normal browser-like user agent:

curl -i -A "Mozilla/5.0" https://app.example.com/login

Expected output shape is 200, 302, or your app’s normal login response, but not 403.

Verify it works

Run these checks end to end:

curl -I https://api.example.com/api/ping
curl -I https://app.example.com/login
seq 1 30 | xargs -I{} -P15 curl -s -o /dev/null -w "%{http_code}\n" https://api.example.com/api/ping | sort | uniq -c
curl -i -A "curl/8.7.1" https://app.example.com/login
sudo tail -n 20 /var/log/nginx/access.log

Expected results:

• Normal single requests return your usual 200, 204, or 302
• Burst test includes 429
• Obvious scripted user agents on login return 403 if you enabled the UA block
• Access logs show distinct client IPs in $remote_addr/$realip_remote_addr, not only the proxy IP

A misconfigured real IP setup usually looks like this in logs, where every request shares one address:

198.51.100.7 realip=198.51.100.7 xff="203.0.113.44" [30/Oct/2026:12:10:01 +0000] "GET /api/ping HTTP/2.0" 429 169
198.51.100.7 realip=198.51.100.7 xff="203.0.113.45" [30/Oct/2026:12:10:02 +0000] "GET /api/ping HTTP/2.0" 429 169

Correct behavior looks like this, where the client IP is restored:

203.0.113.44 realip=203.0.113.44 xff="203.0.113.44, 198.51.100.7" [30/Oct/2026:12:11:01 +0000] "GET /api/ping HTTP/2.0" 200 612
203.0.113.45 realip=203.0.113.45 xff="203.0.113.45, 198.51.100.7" [30/Oct/2026:12:11:02 +0000] "GET /api/ping HTTP/2.0" 200 612

Common pitfalls

All users get 429 after enabling limits

Mistake: rate limiting on $binary_remote_addr without restoring the real client IP from the proxy. Symptom: every request appears from the same CDN/load balancer IP in logs; many legitimate users get 429 immediately. Fix: add literal set_real_ip_from <proxy-cidr>; real_ip_header X-Forwarded-For; real_ip_recursive on; and reload nginx.

You trusted the whole internet for X-Forwarded-For

Mistake: set_real_ip_from 0.0.0.0/0; or equivalent broad trust. Symptom: attackers spoof X-Forwarded-For and bypass per-IP limits. Fix: replace broad trust with only your actual proxy/CDN CIDRs.

Login page breaks for password managers or SSO callbacks

Mistake: applying bot blocks or strict per-minute limits to callback endpoints or POST-heavy auth flows. Symptom: intermittent 403 or 429 during login, especially after MFA or OAuth redirects. Fix: limit only /login and token endpoints you own; exempt /oauth/callback, /saml/acs, and health endpoints explicitly.

CDN and nginx both rate-limit, causing harsher throttling than expected

Mistake: setting the same threshold at both layers without accounting for retries and bursts. Symptom: users hit 429 sooner than your math suggests; logs show edge and origin both rejecting. Fix: keep the stricter limit at the outermost edge and set nginx slightly higher or only as a fallback.

HTTP/2 multiplexing makes your test look inconsistent

Mistake: using a single keepalive client and assuming one TCP connection equals one request stream behavior. Symptom: burst tests vary between runs, especially with browser-based testing. Fix: use the parallel xargs -P curl command shown above and judge by aggregate 200/429 counts, not one request sequence.

Blocking by User-Agent catches harmless scripts but misses real bots

Mistake: relying only on UA regexes. Symptom: your own monitoring script gets 403, while browser-imitating bots still hit the app. Fix: keep UA blocking only on login paths and pair it with per-IP rate limits or your CDN/WAF’s bot/challenge signal at the edge.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.