Fix mTLS handshake failures after rotating service certificates

TL;DR — If mTLS starts failing right after a certificate rotation, the most likely cause is that one side no longer trusts the new issuing CA or is still presenting an old cert/key pair. Start with openssl s_client -connect host:port -servername host -showcerts -state -msg -cert client.crt -key client.key -CAfile ca.pem and compare the presented chain, SANs, EKU, and expiry against what the peer expects. In most incidents, fixing the trust bundle on the verifier or reloading the workload so it picks up the new secret restores traffic. Reading time: ~6 min

The scenario

You rotate internal service certificates on a Tuesday afternoon because the old intermediate expires tonight. The deploy finishes, Kubernetes says the pods are Ready, and your dashboards still show green CPU and memory. Ten minutes later, checkout cannot call payments, the retry queue starts climbing, and every service-to-service hop protected by mTLS is throwing handshake errors. You need to answer one question fast: is the client presenting the wrong cert, or is the server rejecting a cert it should trust?

Symptoms

• Sudden spike in 5xx between internal services immediately after cert rotation.
• curl from one service to another fails with:

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to payments.internal:8443

• Or:

curl: (60) SSL certificate problem: unable to get local issuer certificate

• openssl s_client shows verifier errors such as:

verify error:num=20:unable to get local issuer certificate
verify error:num=21:unable to verify the first certificate
Verify return code: 21 (unable to verify the first certificate)

• Server-side logs from nginx / Envoy / app TLS stack contain lines like:

SSL_do_handshake() failed (SSL: error:0A000086:SSL routines::certificate verify failed)
peer did not return a certificate
tls: failed to verify certificate: x509: certificate signed by unknown authority
remote error: tls: bad certificate
x509: certificate is valid for api.internal, not payments.internal
x509: certificate specifies an incompatible key usage
x509: certificate has expired or is not yet valid

• gRPC clients report:

rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"

• Java clients report:

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Likely causes

Step-by-step diagnosis

1. Check the exact verifier error from the failing side.

curl -vk --cert /etc/tls/client.crt --key /etc/tls/client.key --cacert /etc/tls/ca.pem https://payments.internal:8443/health

If you see unknown authority, unable to get local issuer certificate, or unable to verify the first certificate, jump to Fixes → Verifier trust store does not include the new root/intermediate CA or Fixes → Full chain is not being presented by the server. If you see certificate is valid for ... not ..., jump to SAN mismatch. If you see incompatible key usage, jump to EKU/Key Usage.

1. Inspect what the server is actually presenting on the wire.

openssl s_client -connect payments.internal:8443 -servername payments.internal -showcerts -state -brief </dev/null

This is your problem if the leaf serial/fingerprint does not match the newly issued cert, or if only the leaf is shown and the intermediate is missing. Jump to stale cert/key or full-chain fix.

Typical bad output shape:

CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Peer certificate: CN = payments.internal
Verification error: unable to verify the first certificate
---
Certificate chain
 0 s:CN = payments.internal
   i:CN = Internal Intermediate CA 2026
---
Verify return code: 21 (unable to verify the first certificate)

1. Verify the new cert against the CA bundle the peer uses.

openssl verify -verbose -CAfile /etc/ssl/mtls/ca-bundle.pem /etc/tls/client.crt

If output is not /etc/tls/client.crt: OK, this is your problem. Typical failures:

error 20 at 0 depth lookup: unable to get local issuer certificate
error 10 at 0 depth lookup: certificate has expired
error 79 at 0 depth lookup: invalid CA certificate

Jump to trust-store, clock, or chain fix depending on the error.

1. Check SANs and EKU on the rotated certs.

openssl x509 -in /etc/tls/server.crt -noout -text | sed -n '/Subject:/p;/Subject Alternative Name/,+1p;/Extended Key Usage/,+1p;/Key Usage/,+1p'

This is your problem if the SAN does not include the DNS name clients dial (payments.internal) or if EKU lacks TLS Web Server Authentication for server certs or TLS Web Client Authentication for client certs. Jump to SAN mismatch or EKU fix.

1. Confirm the private key matches the certificate.

openssl x509 -noout -modulus -in /etc/tls/tls.crt | openssl md5 && openssl rsa -noout -modulus -in /etc/tls/tls.key | openssl md5

If the hashes differ, the mounted key does not belong to the cert. Jump to stale cert/key fix.

1. Check time validity from the failing host.

date -u
openssl x509 -in /etc/tls/tls.crt -noout -startdate -enddate

If current UTC is before notBefore or after notAfter, jump to clock skew fix.

1. If running in Kubernetes, verify the Secret changed and the pod actually loaded it.

kubectl get secret payments-mtls -o jsonpath='{.metadata.resourceVersion}{"\n"}{.data.tls\.crt}' | head -c 80; echo
kubectl exec deploy/payments -- sha -256sum /etc/tls/tls.crt /etc/tls/tls.key

If the Secret updated but the file in the pod or the on-wire cert did not, jump to stale cert/key fix.

[Client Service] ---- 1. present client cert ----> [Server Service]
      |                                            |
      v                                            v
[Client trust bundle]                        [Server trust bundle]
      | 2. verify server chain                    | 3. verify client chain
      v                                            v
[Root/Intermediate CA] <---- 4. issued certs ---- [Internal PKI]

Fixes

Verifier trust store does not include the new root/intermediate CA

Update the CA bundle used for mTLS verification on the side rejecting the peer. Include both old and new intermediates during overlap if you are doing phased rotation.

Linux file-based trust bundle:

cat old-intermediate.pem new-intermediate.pem root.pem > /etc/ssl/mtls/ca-bundle.pem
openssl verify -CAfile /etc/ssl/mtls/ca-bundle.pem /etc/tls/client.crt
systemctl reload nginx || systemctl restart nginx

nginx mTLS verifier:

ssl_client_certificate /etc/ssl/mtls/ca-bundle.pem;
ssl_verify_client on;
ssl_verify_depth 3;

Envoy example:

transport_socket:
  name: envoy.transport_sockets.tls
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
    common_tls_context:
      validation_context:
        trusted_ca:
          filename: /etc/ssl/mtls/ca-bundle.pem

Verify it worked:

openssl verify -CAfile /etc/ssl/mtls/ca-bundle.pem /etc/tls/client.crt

Workload is still serving the old cert or mismatched cert/key after secret rotation

Reload or restart the process so it reopens cert files, and fix any cert/key mismatch.

⚠️ Restarting a busy ingress or service can cause a short traffic blip if you do not have multiple replicas.

Kubernetes rollout restart:

kubectl rollout restart deploy/payments
kubectl rollout status deploy/payments

systemd service reload/restart:

systemctl reload nginx || systemctl restart nginx

If cert/key mismatch is confirmed, replace both from the same issuance event:

install -m 600 new-server.key /etc/tls/tls.key
install -m 644 new-server-fullchain.crt /etc/tls/tls.crt
openssl x509 -noout -modulus -in /etc/tls/tls.crt | openssl md5
openssl rsa -noout -modulus -in /etc/tls/tls.key | openssl md5

Verify it worked:

openssl s_client -connect payments.internal:8443 -servername payments.internal </dev/null 2>/dev/null | openssl x509 -noout -serial -fingerprint -sha256

New cert SAN/CN does not match the service name clients use

Reissue the cert with the exact DNS names clients dial. Do not rely on CN-only matching.

OpenSSL CSR config example:

[ req ]
distinguished_name = dn
req_extensions = req_ext
prompt = no
[ dn ]
CN = payments.internal
[ req_ext ]
subjectAltName = @alt_names
extendedKeyUsage = serverAuth
[ alt_names ]
DNS.1 = payments.internal
DNS.2 = payments
DNS.3 = payments.svc.cluster.local

Generate CSR:

openssl req -new -key /etc/tls/tls.key -out payments.csr -config csr.conf

After issuing, deploy the new cert and reload the service. Verify it worked:

openssl x509 -in /etc/tls/tls.crt -noout -text | grep -A1 "Subject Alternative Name"

Extended Key Usage / Key Usage is wrong for mTLS client or server auth

Reissue with the correct EKU. For mTLS, server certs need serverAuth; client certs need clientAuth. Some stacks require both when the same cert is reused on both sides; that works, but split certs are cleaner.

OpenSSL extension snippets:

extendedKeyUsage = clientAuth
keyUsage = digitalSignature, keyEncipherment

extendedKeyUsage = serverAuth
keyUsage = digitalSignature, keyEncipherment

Deploy the corrected cert, then reload/restart the process. Verify it worked:

openssl x509 -in /etc/tls/tls.crt -noout -text | grep -A3 "Extended Key Usage"

Full chain is not being presented by the server

Configure the server to present leaf + intermediate(s), not just the leaf.

nginx:

ssl_certificate     /etc/tls/fullchain.pem;
ssl_certificate_key /etc/tls/tls.key;

Build fullchain.pem in the right order:

cat server.crt intermediate.pem > /etc/tls/fullchain.pem
nginx -t && systemctl reload nginx

For apps that separate leaf and chain, point the chain file explicitly if supported; otherwise concatenate as above. Verify it worked:

openssl s_client -connect payments.internal:8443 -servername payments.internal -showcerts </dev/null | sed -n '/Certificate chain/,$p'

System clocks skewed enough that new cert is "not yet valid"

Fix NTP/chrony on the failing host, then retry the handshake.

chrony:

chronyc tracking
chronyc sources -v
systemctl restart chronyd || systemctl restart chrony

Quick one-off check against a reference host:

date -u
ssh ntp-reference 'date -u'

Verify it worked:

date -u && openssl x509 -in /etc/tls/tls.crt -noout -startdate -enddate

TLS secret mounted but process never reloaded, so it still uses stale files in memory

Some runtimes watch files; many do not. If the secret volume changed but the process still serves the old cert, force a reload on secret change.

Kubernetes annotation pattern to trigger rollout on secret checksum:

spec:
  template:
    metadata:
      annotations:
        checksum/tls-secret: "sha256:RENDERED_BY_CI"

Then redeploy so any secret change updates the pod template hash.

For nginx, use a reload hook after secret refresh:

nginx -t && nginx -s reload

Verify it worked:

openssl s_client -connect payments.internal:8443 -servername payments.internal </dev/null 2>/dev/null | openssl x509 -noout -dates -serial

Prevention

• Ship overlapping trust bundles during CA rotation. Keep old and new intermediates trusted for at least one deploy cycle.

cat old-int.pem new-int.pem root.pem > ca-bundle.pem
openssl verify -CAfile ca-bundle.pem new-server.crt
openssl verify -CAfile ca-bundle.pem old-server.crt

• Add a CI check that fails cert artifacts missing SAN or EKU.

openssl x509 -in dist/server.crt -noout -text | grep -q "DNS:payments.internal"
openssl x509 -in dist/server.crt -noout -text | grep -q "TLS Web Server Authentication"
openssl x509 -in dist/client.crt -noout -text | grep -q "TLS Web Client Authentication"

• Add a pre-deploy smoke test that performs a real mTLS handshake from the same image/runtime your service uses.

curl -fsSvk --cert dist/client.crt --key dist/client.key --cacert dist/ca-bundle.pem https://payments.internal:8443/health

• Alert on certificate expiry and on sudden increases in TLS handshake failures, not just HTTP 5xx. Scrape logs for these substrings: certificate verify failed, unknown authority, bad certificate, not yet valid.

{"match":["certificate verify failed","unknown authority","bad certificate","not yet valid"]}

• Record and compare on-wire cert fingerprints after rotation. A post-deploy job should fail if the served fingerprint is not the newly issued one.

EXPECTED="SHA256 Fingerprint=AA:BB:CC:..."
ACTUAL=$(openssl s_client -connect payments.internal:8443 -servername payments.internal </dev/null 2>/dev/null | openssl x509 -noout -fingerprint -sha256)
test "$ACTUAL" = "$EXPECTED"

• If you use Kubernetes Secrets for TLS, tie pod rollout to secret content hash so cert updates cannot sit unused in mounted volumes.

metadata:
  annotations:
    checksum/tls-secret: "{{ .Values.tlsSecretSha256 }}"

This article was written by an AI system and published pending human review. Verify anything you intend to act on.