JWKS rotation breaks JWT validation: fix kid mismatches and cache staleness

TL;DR — If JWT validation starts failing right after key rotation, the most common cause is that your verifier is still using a cached JWKS that does not contain the new kid. Confirm by comparing the token header kid with the live JWKS, then either force-refresh the verifier cache or roll keys safely by publishing the new key in JWKS before signing tokens with it and keeping the old key available until all old tokens expire. Reading time: ~6 min

The scenario

It is Tuesday afternoon, you rotate your IdP signing key, deploy looks normal, and five minutes later your API starts returning 401s for a slice of users. Existing sessions still work in one service, but a newer deploy or a different region is rejecting the same bearer token with invalid signature or unable to find a signing key that matches. Your auth server says the new key is active, your JWKS endpoint looks healthy in the browser, and now you need to prove whether this is cache staleness, a kid mismatch, or a bad rotation order before you start bouncing pods.

[Client] ---- 1. Bearer token ----> [API]
   |                                  |
   |                                  v
   |                        [JWT verifier cache]
   |                                  |
   |                                  v
   |                        [JWKS endpoint / .well-known]
   |                                  |
   |                                  v
   ^---- 5. 401 invalid_token ---- [IdP signing keys]

Symptoms

• API returns 401 Unauthorized or 403 Forbidden immediately after key rotation.
• Verifier logs contain one of these exact shapes:

JsonWebKeySetsError: The JWKS endpoint did not contain any keys
JsonWebTokenError: invalid signature
JWKSError: Unable to find a signing key that matches 'kid "a1b2c3"'
JOSEError: no applicable key found in the JSON Web Key Set
IDX10501: Signature validation failed. Unable to match key: kid: 'a1b2c3'
SecurityTokenSignatureKeyNotFoundException: IDX10501: Signature validation failed

• One service validates tokens, another rejects the same token.
• Freshly issued tokens fail, older tokens still pass; or the reverse during rollback.
• curl to JWKS returns a redirect, HTML, or stale cache headers instead of JSON:

curl -i https://id.example.com/.well-known/jwks.json

HTTP/2 301
location: /login
content-type: text/html; charset=utf-8
cache-control: public, max-age=3600

• Token header kid is not present in the live JWKS.
• Failures cluster by region/pod because some instances refreshed JWKS and others did not.

Likely causes

Step-by-step diagnosis

1. Decode the failing token header and capture kid, alg, iss.

TOKEN='eyJ...'
printf '%s' "$TOKEN" | cut -d. -f1 | tr '_-' '/+' | base64 -d 2>/dev/null | jq .

This is your problem if kid is null, empty, or unexpected for your current signer. Jump to Fixes → Missing or mismatched kid in signed tokens.

1. Compare the token kid with the live JWKS.

KID=$(printf '%s' "$TOKEN" | cut -d. -f1 | tr '_-' '/+' | base64 -d 2>/dev/null | jq -r .kid)
curl -fsS https://id.example.com/.well-known/jwks.json | jq -r --arg KID "$KID" '.keys[]?.kid' | grep -Fx "$KID" && echo FOUND || echo MISSING

If output is MISSING, either the new key was not published before use or you are hitting the wrong JWKS. Jump to Fixes → New key not published in JWKS before use or Fixes → Wrong issuer/JWKS URL.

1. Check whether the JWKS endpoint is actually serving JSON without redirects or auth.

curl -iL https://id.example.com/.well-known/jwks.json

This is your problem if you see 301, 302, 401, 403, HTML content, or content-type: text/html. A healthy response looks like:

HTTP/2 200
content-type: application/json
cache-control: public, max-age=300
etag: "9f1c..."

{"keys":[{"kty":"RSA","kid":"a1b2c3",...}]}

Jump to Fixes → JWKS endpoint redirect/auth/cache misconfiguration.

1. Check whether your verifier process has cached JWKS longer than your rotation window.

curl -sI https://id.example.com/.well-known/jwks.json | egrep -i 'cache-control|etag|last-modified|age'

If max-age is large, or your app/library caches keys in memory and only refreshes on interval/restart, this is likely your problem. Typical sign: one pod works after restart, another still fails. Jump to Fixes → Stale verifier JWKS cache.

1. Confirm whether old tokens still need the retired key.

curl -fsS https://id.example.com/.well-known/jwks.json | jq -r '.keys[]?.kid'

If the old kid is gone but clients still present tokens signed with it and those tokens are not expired yet, this is your problem. Jump to Fixes → Old key removed too early.

1. Validate issuer metadata if rotation coincided with domain or IdP changes.

curl -fsS https://id.example.com/.well-known/openid-configuration | jq -r '.issuer,.jwks_uri'

If your app is configured with a different issuer or stale jwks_uri, jump to Fixes → Wrong issuer/JWKS URL.

Fixes

Stale verifier JWKS cache

Reduce or invalidate the verifier cache, then force a refresh.

Common places to change:

location = /.well-known/jwks.json {
    add_header Cache-Control "public, max-age=300, must-revalidate" always;
    add_header Content-Type "application/json" always;
}

If your app caches JWKS in-process, restart the deployment after lowering cache TTL.

⚠️ Restarting all verifier instances can cause a brief auth spike or outage if done all at once. Roll pods gradually.

kubectl rollout restart deploy/api
kubectl rollout status deploy/api

If you front JWKS with a CDN/reverse proxy, purge that path in your provider dashboard (e.g. CDN cache purge for /.well-known/jwks.json) or bypass cache for that route.

Verify it worked:

for p in $(kubectl get pod -l app=api -o name); do echo "== $p =="; kubectl logs "$p" --tail=50 | egrep 'kid|jwks|signature'; done

Missing or mismatched kid in signed tokens

Publish a stable kid in the signer config and ensure the signer uses the same key material as JWKS.

Example JWT header should look like:

{"alg":"RS256","kid":"a1b2c3","typ":"JWT"}

If your signer is building tokens without kid, fix that in the signing code/config and redeploy. If it uses the wrong kid, regenerate JWKS from the exact public key matching the private key in use; do not hand-edit kid unless your stack derives it consistently.

Quick sanity check with OpenSSL for an RSA keypair:

openssl rsa -in private.pem -pubout -outform PEM > derived-public.pem
openssl pkey -pubin -in derived-public.pem -text -noout | head

Then rebuild the JWKS from derived-public.pem using your normal tooling and publish it.

Verify it worked:

NEW='eyJ...'; printf '%s' "$NEW" | cut -d. -f1 | tr '_-' '/+' | base64 -d 2>/dev/null | jq -r .kid
curl -fsS https://id.example.com/.well-known/jwks.json | jq -r '.keys[]?.kid'

New key not published in JWKS before use

Safe order is: publish new public key in JWKS, wait for caches to age out, then start signing with the new private key, then remove the old key only after old tokens expire.

Recommended sequence:

[IdP] ---- 1. add new public key ----> [JWKS]
[JWKS] ---- 2. cache TTL elapses ----> [Verifiers]
[IdP] ---- 3. sign new tokens w/new kid ----> [Clients]
[Old tokens] ---- 4. expire naturally ----> [APIs]
[IdP] ---- 5. remove old public key ----> [JWKS]

If you already broke production, fastest recovery is to republish the missing public key under the exact kid currently in failing tokens.

Verify it worked:

KID=$(printf '%s' "$TOKEN" | cut -d. -f1 | tr '_-' '/+' | base64 -d 2>/dev/null | jq -r .kid)
curl -fsS https://id.example.com/.well-known/jwks.json | jq -r --arg K "$KID" '.keys[]?.kid' | grep -Fx "$KID" && echo ok

JWKS endpoint redirect/auth/cache misconfiguration

Serve JWKS directly at a stable URL with 200 OK, application/json, and no login/auth redirect.

Bad:

HTTP/2 302
location: /login

Good nginx example:

location = /.well-known/jwks.json {
    default_type application/json;
    add_header Cache-Control "public, max-age=300, must-revalidate" always;
    try_files /jwks.json =404;
}

If a WAF or auth proxy protects all paths by default, add an allow rule for /.well-known/jwks.json and /.well-known/openid-configuration in your provider dashboard or reverse-proxy config.

Verify it worked:

curl -i https://id.example.com/.well-known/jwks.json | sed -n '1,10p'

Old key removed too early

Republish the old public key in JWKS immediately and keep it until every token signed with it is expired plus clock skew.

To estimate the hold period, use your longest token lifetime, not your average. If access tokens live 15 minutes and refresh-token exchange can mint access tokens from older sessions, account for that path too.

⚠️ Do not delete and recreate the old key with a different kid. Existing tokens reference the original kid and will still fail.

Verify it worked:

curl -fsS https://id.example.com/.well-known/jwks.json | jq -r '.keys[]?.kid' | grep -Fx 'old-kid-value'

Wrong issuer/JWKS URL

Point verifiers at the issuer metadata, not a copied JWKS URL from an old environment.

Check and update config such as:

auth:
  issuer: https://id.example.com/
  jwks_uri: https://id.example.com/.well-known/jwks.json

Prefer deriving jwks_uri from /.well-known/openid-configuration where your library supports it, so issuer and keys move together.

Verify it worked:

curl -fsS https://id.example.com/.well-known/openid-configuration | jq -r '.issuer,.jwks_uri'

Prevention

• Add a CI smoke test that fails a rotation if the new signer kid is not present in JWKS before deploy:

NEW_KID=$(jq -r .kid build/token-header.json)
curl -fsS https://id.example.com/.well-known/jwks.json | jq -e --arg K "$NEW_KID" '.keys[] | select(.kid==$K)' >/dev/null

• Pin short JWKS cache headers at the origin and bypass CDN caching for the well-known paths:

location ~ ^/\.well-known/(jwks\.json|openid-configuration)$ {
    add_header Cache-Control "public, max-age=300, must-revalidate" always;
}

• Alert on verifier log patterns that indicate rotation breakage:

"Unable to find a signing key"
"IDX10501"
"no applicable key found"
"invalid signature"

• Run a pre-rotation overlap window checklist in automation: add new key to JWKS, wait max-age + verifier_refresh_interval, then switch signer. Encode the wait explicitly in your pipeline/runbook.

sleep 600  # example: 300s CDN/origin max-age + 300s app refresh interval

• Keep old keys published for max_token_ttl + clock_skew + rollout_buffer. Example policy:

rotation:
  jwks_overlap: 2h
  clock_skew: 5m
  access_token_ttl: 15m

• Add a canary validation check from each region/pod pool after rotation:

for url in https://api-us.example.com/health/auth https://api-eu.example.com/health/auth; do curl -fsS "$url" || echo "FAIL $url"; done

This article was written by an AI system and published pending human review. Verify anything you intend to act on.