Set up mTLS between services with short-lived workload certificates

TL;DR — You will stand up a private CA, issue a server cert and a client/workload cert with a short TTL, and configure nginx plus curl so requests only succeed when both sides present valid certificates. The most common failure is SAN or trust-chain mismatch: issue certs with the exact DNS names in subjectAltName, and point both sides at the CA certificate they should trust. Reading time: ~5 min

Goal

When you finish, one service endpoint accepts traffic only from clients presenting a valid short-lived workload certificate signed by your private CA, and you can prove it with curl returning 200 with the client cert and a TLS failure or 400/403/496-style rejection without it.

Prerequisites

• Shell access on a Linux or macOS machine with bash, openssl >= 3.0 — check with:

openssl version

• nginx >= 1.22 installed on the server that will require client certificates — check with:

nginx -v

• curl >= 8.0 with TLS support — check with:

curl --version

• systemctl access or equivalent to reload nginx
• A DNS name for the server, for example api.internal.example.com, resolving to the nginx host
• TCP/443 reachable from the client to the server
• Permission to write files under /etc/nginx/mtls on the server and a working directory on the client
• Time synchronized on both hosts (timedatectl status should not show large drift)

Steps

Step 1: Create a private CA and issue short-lived certificates

Run these commands on a secure admin machine. This creates one CA, one server certificate for api.internal.example.com, and one client certificate for payments-worker valid for 24 hours.

⚠️ The CA private key created here can mint trusted workload certificates. Store ca.key offline or in an HSM/KMS-backed workflow after this test setup. Do not leave it on the nginx host.

mkdir -p ~/mtls-demo && cd ~/mtls-demo
umask 077

openssl genpkey -algorithm RSA -out ca.key -pkeyopt rsa_keygen_bits:4096
openssl req -x509 -new -key ca.key -sha256 -days 3650 \
  -out ca.crt \
  -subj "/CN=internal-workload-ca"

cat > server-openssl.cnf <<'EOF'
[req]
distinguished_name = dn
req_extensions = req_ext
prompt = no
[dn]
CN = api.internal.example.com
[req_ext]
subjectAltName = @alt_names
extendedKeyUsage = serverAuth
keyUsage = digitalSignature,keyEncipherment
[alt_names]
DNS.1 = api.internal.example.com
EOF

openssl genpkey -algorithm RSA -out server.key -pkeyopt rsa_keygen_bits:2048
openssl req -new -key server.key -out server.csr -config server-openssl.cnf
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out server.crt -days 7 -sha256 -copy_extensions copy

cat > client-openssl.cnf <<'EOF'
[req]
distinguished_name = dn
req_extensions = req_ext
prompt = no
[dn]
CN = payments-worker
[req_ext]
extendedKeyUsage = clientAuth
keyUsage = digitalSignature
subjectAltName = @alt_names
[alt_names]
DNS.1 = payments-worker
EOF

openssl genpkey -algorithm RSA -out client.key -pkeyopt rsa_keygen_bits:2048
openssl req -new -key client.key -out client.csr -config client-openssl.cnf
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client.crt -days 1 -sha256 -copy_extensions copy

openssl x509 -in server.crt -noout -text | grep -A2 "Subject Alternative Name"
openssl x509 -in client.crt -noout -dates

Success looks like SAN output containing DNS:api.internal.example.com and client cert dates showing notAfter about 24 hours ahead.

Step 2: Install the server certificate and require client certificates in nginx

Copy server.crt, server.key, and ca.crt to the nginx host, then write this exact server block.

sudo mkdir -p /etc/nginx/mtls
sudo cp server.crt server.key ca.crt /etc/nginx/mtls/
sudo chmod 600 /etc/nginx/mtls/server.key

sudo tee /etc/nginx/conf.d/mtls-api.conf >/dev/null <<'EOF'
server {
    listen 443 ssl;
    server_name api.internal.example.com;

    ssl_certificate           /etc/nginx/mtls/server.crt;
    ssl_certificate_key       /etc/nginx/mtls/server.key;
    ssl_client_certificate    /etc/nginx/mtls/ca.crt;
    ssl_verify_client         on;
    ssl_verify_depth          2;
    ssl_protocols             TLSv1.2 TLSv1.3;

    location /healthz {
        add_header Content-Type text/plain;
        return 200 "ok\n";
    }

    location /whoami {
        add_header Content-Type text/plain;
        return 200 "client_dn=$ssl_client_s_dn\nverify=$ssl_client_verify\n";
    }
}
EOF

sudo nginx -t && sudo systemctl reload nginx

Success looks like nginx: configuration file ... test is successful and no reload error from systemctl.

Step 3: Verify the server rejects requests without a client certificate

From the client machine, trust the CA for server verification but do not send a client certificate yet.

curl -v --cacert ca.crt https://api.internal.example.com/healthz

Success for this step is a rejection. Common output shapes are one of these:

* TLSv1.3 (IN), TLS handshake, Request CERT (13):
...
< HTTP/1.1 400 Bad Request
< Server: nginx
< Content-Type: text/html

or a TLS alert such as:

curl: (56) OpenSSL SSL_read: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0

Step 4: Call the service with the workload certificate

Now send the client certificate and key.

curl -v \
  --cacert ca.crt \
  --cert client.crt \
  --key client.key \
  https://api.internal.example.com/whoami

Success looks like HTTP/1.1 200 OK and a body similar to:

client_dn=CN=payments-worker
verify=SUCCESS

Step 5: Package the workload cert for applications that expect PKCS#12

Some runtimes want a .p12 bundle instead of separate PEM files.

openssl pkcs12 -export \
  -out client.p12 \
  -inkey client.key \
  -in client.crt \
  -certfile ca.crt \
  -passout pass:changeit

Success looks like no output and exit code 0. The file client.p12 exists.

Step 6: Rotate the client certificate before expiry

This is the short-lived-cert part: issue a replacement cert, switch the client to it, then retire the old one. Reuse the same CSR if identity stays the same.

openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client-next.crt -days 1 -sha256 -copy_extensions copy

curl -sS \
  --cacert ca.crt \
  --cert client-next.crt \
  --key client.key \
  https://api.internal.example.com/healthz

Success looks like the response body:

ok

Verify it works

Run all three checks end to end.

# 1) Server cert chains to your CA and presents the expected name
openssl s_client -connect api.internal.example.com:443 \
  -servername api.internal.example.com \
  -CAfile ca.crt </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer

# 2) No client cert: request is rejected
curl -sS -o /dev/null -w "%{http_code}\n" --cacert ca.crt \
  https://api.internal.example.com/healthz || true

# 3) Valid client cert: request succeeds
curl -sS --cacert ca.crt --cert client.crt --key client.key \
  https://api.internal.example.com/whoami

Expected results:

subject=CN = api.internal.example.com
issuer=CN = internal-workload-ca
400
client_dn=CN=payments-worker
verify=SUCCESS

If check 2 returns 000, that is still a failure without a client cert; curl hit a TLS-level error before HTTP.

[Client] ---- 1. TLS ClientHello ----> [nginx API]
[Client] <---- 2. CertificateRequest ---- [nginx API]
[Client] ---- 3. client.crt + proof ----> [nginx API]
[Client] <---- 4. 200 /whoami ----------- [nginx API]

Common pitfalls

SAN missing or wrong DNS name

Mistake: issuing the server cert with only CN=api.internal.example.com and no subjectAltName, or with the wrong hostname. Symptom: curl fails with SSL: no alternative certificate subject name matches target host name 'api.internal.example.com'. Fix: reissue server.crt with DNS.1 = api.internal.example.com in the OpenSSL config and reload nginx.

Trusting the wrong CA file on either side

Mistake: pointing nginx ssl_client_certificate at the server cert instead of the CA cert, or using the system CA bundle in curl when your private CA is not installed there. Symptom: nginx error log shows client SSL certificate verify error: (20:unable to get local issuer certificate) or curl: (60) SSL certificate problem: unable to get local issuer certificate. Fix: set nginx ssl_client_certificate /etc/nginx/mtls/ca.crt; and call curl with --cacert ca.crt.

Client certificate missing extendedKeyUsage = clientAuth

Mistake: issuing a generic cert without client auth EKU. Symptom: handshake fails; nginx may log unsupported certificate purpose, and curl -v ends with a TLS alert after sending the cert. Fix: reissue the client cert with extendedKeyUsage = clientAuth exactly as shown in client-openssl.cnf.

Certificate expired because rotation was tested too late

Mistake: using a 24-hour cert and validating the next day without reissuing. Symptom: curl shows certificate has expired; nginx logs client SSL certificate verify error: (10:certificate has expired). Fix: mint a new client cert with the Step 6 command and switch the client to client-next.crt.

Clock skew between hosts

Mistake: server or client clock is off by minutes or hours. Symptom: a freshly issued cert fails with certificate is not yet valid. Fix: sync time on both hosts with your normal NTP service, then retry; on systemd hosts check with timedatectl status.

Forgetting SNI when testing with openssl s_client

Mistake: testing a name-based virtual host without -servername. Symptom: openssl s_client shows the wrong certificate, while curl to the hostname behaves differently. Fix: include -servername api.internal.example.com in the openssl s_client command.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.