429s from Model APIs: Read Rate-Limit Headers and Back Off Correctly

TL;DR — If your model calls start returning 429 Too Many Requests, do not guess. Capture the response headers, look for Retry-After and provider-specific rate-limit headers, then back off on the dimension you actually exhausted: requests, tokens, or concurrent in-flight calls. The most common fix is to honor Retry-After, add jittered exponential backoff, and cap concurrency with a local semaphore so retries do not amplify the spike. Reading time: ~6 min

The scenario

It is Tuesday at 14:37. You ship a feature that fans out one user action into several model calls, traffic is only slightly above normal, and suddenly your app starts logging 429 errors in bursts. The provider status page is green, your network looks fine, and retries in your HTTP client are making things worse instead of better. Product is asking whether this is a provider outage; the real question is whether you are exhausting request rate, token rate, or concurrency and how to stop the retry storm.

Symptoms

• API responses with HTTP 429 Too Many Requests.
• Error bodies like:

{"error":{"message":"Rate limit exceeded","type":"rate_limit_error","code":"rate_limit_exceeded"}}

• Or:

{"error":{"message":"Too many requests, please retry later.","type":"throttling_error"}}

• Response headers containing one or more of:

Retry-After: 17
X-RateLimit-Limit-Requests: 300
X-RateLimit-Remaining-Requests: 0
X-RateLimit-Reset-Requests: 1723298452
X-RateLimit-Limit-Tokens: 90000
X-RateLimit-Remaining-Tokens: 120
X-RateLimit-Reset-Tokens: 1723298460

• Application logs showing retries clustered at the same second:

POST /v1/responses -> 429 in 412ms req_id=abc123 attempt=1
POST /v1/responses -> 429 in 405ms req_id=abc124 attempt=2 sleep_ms=500
POST /v1/responses -> 429 in 398ms req_id=abc125 attempt=3 sleep_ms=1000

• Queue depth rising while successful throughput drops.
• End users seeing slow responses, partial failures, or generic "please try again" messages.
• If you are saturating concurrency rather than rate, you may see low request volume but many in-flight requests and long tail latency.

Likely causes

Step-by-step diagnosis

1. Capture one throttled response with headers.

curl -sS -D /tmp/headers.txt -o /tmp/body.json \
  https://api.provider.example/v1/endpoint \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d @req.json; echo $?; echo; sed -n '1,40p' /tmp/headers.txt; echo; cat /tmp/body.json

If you see HTTP/1.1 429 plus Retry-After or any X-RateLimit-*/RateLimit-* headers, this is your problem. Jump to the matching fix: Retry-After ignored, token quota exhausted, or concurrency too high.

1. Determine which dimension is exhausted.

sed -n '/^Retry-After\|^X-RateLimit\|^RateLimit/p' /tmp/headers.txt

Typical shapes:

Retry-After: 12
X-RateLimit-Remaining-Requests: 0
X-RateLimit-Reset-Requests: 1723298452

This means request-rate exhaustion. Or:

X-RateLimit-Remaining-Requests: 42
X-RateLimit-Remaining-Tokens: 0
X-RateLimit-Reset-Tokens: 1723298460

This means token exhaustion even though request count remains. Jump to the corresponding fix.

1. Check whether your client is retrying too early or too often.

grep -nE '429|Retry-After|attempt=|backoff|sleep_ms=' app.log | tail -n 100

If retries happen at fixed intervals like 500ms, 1000ms, 2000ms while the server says Retry-After: 12, your retry policy is wrong. Jump to Client ignores Retry-After.

1. Check for concurrency spikes.

ss -tn state established '( dport = :443 )' | tail -n +2 | wc -l

If established outbound TLS connections spike with the incident and your workers/serverless autoscaling also spiked, you are likely hitting a concurrency limiter or causing self-throttling. Jump to Too many concurrent in-flight requests.

1. Check for retry multiplication across layers.

grep -RniE 'retry|backoff|max_retries|axios-retry|urllib3|fetch-retry|tenacity|resilience|circuit' .

If you find retries in the SDK, your HTTP client, your job worker, and your reverse proxy, you have multiplicative retries. Example: 3 SDK retries × 5 job retries = 15 calls per logical request. Jump to Retries from multiple layers.

1. Validate reset-time parsing and local clock.

date -u; python - <<'PY'
from email.utils import parsedate_to_datetime
import time
print('epoch_now=', int(time.time()))
print('retry_after_seconds=', 12)
print('wake_at=', int(time.time()) + 12)
print(parsedate_to_datetime('Wed, 21 Oct 2015 07:28:00 GMT').timestamp())
PY

If your code treats epoch milliseconds as seconds, or HTTP-date as local time, retries will fire too early. Jump to Clock or reset-time parsing bug.

1. Check whether multiple services share one quota bucket.

for h in app worker cron staging; do echo "[$h]"; ssh $h 'printenv | grep API_KEY'; done

If prod, staging, and background workers all use the same key, one noisy service can starve the rest. Jump to Shared API key across services/environments.

Fixes

Client ignores Retry-After or rate-limit headers and retries too aggressively

Honor Retry-After first. If absent, compute sleep from reset headers; if neither exists, use capped exponential backoff with full jitter.

Node.js example:

function parseRetryAfter(h) {
  if (!h) return null;
  const s = Number(h);
  if (!Number.isNaN(s)) return s * 1000;
  const t = Date.parse(h);
  return Number.isNaN(t) ? null : Math.max(0, t - Date.now());
}

function fullJitterDelay(attempt, baseMs = 500, capMs = 30000) {
  const max = Math.min(capMs, baseMs * 2 ** attempt);
  return Math.floor(Math.random() * max);
}

async function callWithBackoff(doRequest, maxAttempts = 6) {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    const res = await doRequest();
    if (res.status !== 429) return res;
    const ra = parseRetryAfter(res.headers.get('retry-after'));
    const delay = ra ?? fullJitterDelay(attempt);
    await new Promise(r => setTimeout(r, delay));
  }
  throw new Error('rate-limited after maxAttempts');
}

If your SDK auto-retries, reduce or disable one layer so only one component owns retry timing.

Verify it worked:

grep -E '429|sleep_ms=' app.log | tail -n 20

You should see sleeps at or above the server-provided delay and fewer repeated 429s per logical request.

You exhausted token-per-window quota, not request count

Reduce prompt size, cap output tokens, and batch less aggressively. If you estimate tokens client-side, reject or defer oversized work before sending it.

Example request changes:

{
  "model": "your-model",
  "input": "...",
  "max_output_tokens": 256
}

Worker-side token budget gate:

TOKEN_BUDGET_PER_MIN = 80000
if estimated_prompt_tokens + requested_output_tokens > 4000:
    raise ValueError("request too large for current budget")

If you have a queue, drain at a token rate instead of a request rate. A simple leaky bucket keyed on estimated tokens is enough.

Verify it worked:

grep -E 'usage|prompt_tokens|completion_tokens|429' app.log | tail -n 50

You should see lower average tokens per request and 429s disappear while request count may stay similar.

Too many concurrent in-flight requests from workers/serverless instances

Cap concurrency locally so autoscaling does not create a thundering herd.

Node.js semaphore example:

import pLimit from 'p-limit';
const limit = pLimit(8);
const tasks = inputs.map(x => limit(() => callWithBackoff(() => invokeModel(x))));
await Promise.all(tasks);

Python asyncio example:

sem = asyncio.Semaphore(8)
async def guarded_call(payload):
    async with sem:
        return await call_with_backoff(payload)

If you run queue workers, lower worker concurrency or prefetch.

Verify it worked:

ss -tn state established '( dport = :443 )' | tail -n +2 | wc -l

The connection count should flatten, tail latency should improve, and 429 bursts should stop.

Retries from multiple layers are multiplying each other

Pick one retry owner. Disable retries in the SDK or HTTP client if the job runner already retries, or vice versa.

Python requests/urllib3 example:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
retry = Retry(total=0)
session.mount('https://', HTTPAdapter(max_retries=retry))

Job worker example: keep job retries, disable per-request retries except for one short backoff loop around 429.

Verify it worked:

grep -E 'attempt=' app.log | tail -n 50

A single user action should no longer fan out into many duplicate provider calls.

Clock or reset-time parsing bug causes early retries

Treat Retry-After as either delta-seconds or HTTP-date. Treat reset headers as documented by the provider: many use epoch seconds, some use durations. Do not assume milliseconds.

Shell sanity check:

python - <<'PY'
import time
reset = 1723298460
print('sleep_s=', max(0, reset - int(time.time())))
PY

If your code used new Date(reset) on an epoch-seconds value, it interpreted seconds as milliseconds and scheduled nearly immediate retries. Fix by multiplying only when the header is explicitly milliseconds.

Verify it worked:

grep -E 'reset|retry_after|sleep_ms=' app.log | tail -n 20

Sleep values should align with header values instead of near-zero waits.

Shared API key across services/environments is burning the same quota bucket

Split keys by environment and workload. Give production app traffic, background jobs, and staging separate credentials where your provider allows it. At minimum, stop staging and load tests from using the production key.

Environment example:

# prod app
export MODEL_API_KEY=prod_app_key
# prod worker
export MODEL_API_KEY=prod_worker_key
# staging
export MODEL_API_KEY=staging_key

If you cannot split keys, add a local rate limiter per service so one class of traffic cannot consume all capacity.

Verify it worked:

for h in app worker staging; do echo "[$h]"; ssh $h 'printenv | grep MODEL_API_KEY'; done

Keys should differ by environment/workload, and prod 429s should no longer correlate with staging/test activity.

Prevention

• Log rate-limit headers on every non-2xx response and sample them on success.

{"status":429,"retry_after":"12","rl_remaining_requests":"0","rl_remaining_tokens":"120","request_id":"abc123"}

• Add a single shared backoff utility and ban ad hoc retries in code review. In CI, fail if multiple retry libraries are configured:

grep -RniE 'axios-retry|tenacity|urllib3.util.retry|fetch-retry' . && exit 1 || exit 0

• Export metrics for 429_count, retry_count, in_flight_requests, and token usage. Alert on sustained 429s plus rising retries:

alert: ModelProviderThrottling
expr: rate(model_api_429_total[5m]) > 0.1 and rate(model_api_retry_total[5m]) > 0.5
for: 10m

• Put a concurrency cap in config, not code constants scattered across services.

MODEL_API_MAX_CONCURRENCY=8
MODEL_API_MAX_ATTEMPTS=6
MODEL_API_BACKOFF_CAP_MS=30000

• In load tests, simulate provider throttling and verify your client spreads retries with jitter instead of synchronized bursts.

toxiproxy-cli toxic add model_api -t limit_data -a bytes=1024

• If you queue work, rate-limit dequeue by estimated tokens as well as job count.

if token_bucket.try_consume(estimated_tokens):
    process(job)
else:
    requeue(job, delay=5)

That prevents a few giant prompts from exhausting the same budget as many small requests.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.