SPIFFE workload identities across clusters: issue and verify service identity

TL;DR — SPIFFE gives workloads short-lived, cryptographically verifiable identities like spiffe://example.org/ns/payments/sa/api, so services can authenticate each other across clusters without shipping long-lived secrets or maintaining IP-based allowlists. The key design choice is usually this: run one trust domain with federated or shared trust bundles only where cross-cluster calls are real, and verify SPIFFE IDs at the client/server boundary instead of treating mTLS alone as enough. Reading time: ~7 min

What it is and where it sits

SPIFFE workload identity is the identity layer in a zero-trust service-to-service path. It does not replace TLS; it feeds TLS with workload certificates (SVIDs) and a trust bundle so peers can authenticate who is on the other end. In practice, SPIFFE is usually delivered by SPIRE: agents run close to workloads, a server signs identities, and workloads fetch X.509-SVIDs or JWT-SVIDs over a local Workload API socket.

What talks to it:

• Your app, sidecar, or service mesh dataplane reads identities from the Workload API.
• The SPIRE Agent attests the node and asks the SPIRE Server for workload identities.
• The SPIRE Server signs SVIDs and publishes trust bundles.
• Peer services verify the presented certificate chain and the SPIFFE ID in SAN URI.

What it replaces:

• Static client cert distribution by CI/CD or secrets managers.
• Shared API keys between services.
• IP-based identity assumptions like “anything from subnet X is payments”.
• In many cases, mesh-specific identity formats if you want a portable identity layer across clusters or runtimes.

Where it lives in the request flow:

• Before the request: workload gets an SVID from the local agent.
• During TLS handshake: client presents SVID, server validates chain and SPIFFE ID; optionally server presents its own SVID for mutual auth.
• During authorization: app maps SPIFFE ID to policy, e.g. only spiffe://prod.example.org/ns/payments/sa/api may call /settlements/*.

[Workload A] ---- 1. Fetch SVID ----> [SPIRE Agent A]
[SPIRE Agent A] ---- 2. CSR/attest ----> [SPIRE Server]
[SPIRE Server] ---- 3. X.509-SVID ----> [SPIRE Agent A]
[SPIRE Agent A] ---- 4. SVID bundle ----> [Workload A]
[Workload A] ---- 5. mTLS with SVID ----> [Workload B]
[Workload B] ---- 6. Verify chain + ID ----> [AuthZ Policy]

Across clusters, the extra moving part is trust distribution. If Cluster A and Cluster B are in the same trust domain, they need the same root of trust or a synchronized bundle. If they are in different trust domains, they need federation: each side trusts the other domain’s bundle and authorizes the foreign SPIFFE IDs explicitly.

How it actually works

Walk one realistic example: payments-api in cluster prod-eu1 calls ledger in cluster prod-us1. Both run Kubernetes. You want mutual TLS and service identity without copying certs between clusters.

Step 1: Define the identities you want

A common SPIFFE ID shape in Kubernetes is:

• spiffe://prod.example.org/ns/payments/sa/payments-api
• spiffe://prod.example.org/ns/finance/sa/ledger

That identity is not magic; SPIRE issues it because you create registration entries matching workload selectors. In Kubernetes, selectors usually come from pod labels, namespace, and service account.

Step 2: Run SPIRE Server and Agents

Typical topology:

• One SPIRE Server per cluster, or one central server with agents in each cluster.
• One SPIRE Agent DaemonSet per node.
• Kubernetes Workload Registrar or explicit registration entries.

For cross-cluster, the simplest operational model is one trust domain for all production clusters if they are under one security boundary and one platform team. If clusters are owned by different orgs or have different blast-radius requirements, use separate trust domains and federation.

Step 3: Create workload registration entries

On the SPIRE Server, register payments-api and ledger. Example using Kubernetes selectors:

spire-server entry create \
  -spiffeID spiffe://prod.example.org/ns/payments/sa/payments-api \
  -parentID spiffe://prod.example.org/spire/agent/k8s_psat/prod-eu1/node-uid-1234 \
  -selector k8s:ns:payments \
  -selector k8s:sa:payments-api

spire-server entry create \
  -spiffeID spiffe://prod.example.org/ns/finance/sa/ledger \
  -parentID spiffe://prod.example.org/spire/agent/k8s_psat/prod-us1/node-uid-9876 \
  -selector k8s:ns:finance \
  -selector k8s:sa:ledger

In many real deployments you will not hand-create -parentID entries per node; a registrar automates this. The important part is the selectors and the exact SPIFFE ID shape, because your authz policy will depend on it.

Step 4: Workloads fetch SVIDs from the local agent

Inside the pod, the app or sidecar connects to the Workload API Unix socket. With SPIRE tooling, you can inspect what the workload sees:

spire-agent api fetch x509 -socketPath /run/spire/sockets/agent.sock

Typical output shape:

Received 1 SVID(s)
SPIFFE ID:      spiffe://prod.example.org/ns/payments/sa/payments-api
SVID Valid After:  2026-10-30T09:10:11Z
SVID Valid Until:  2026-10-30T10:10:41Z
CA #1 Valid Until: 2031-10-29T00:00:00Z

If the workload is not registered, you usually see an empty result or a permission/attestation error, depending on where it failed. Common shapes:

Error: rpc error: code = PermissionDenied desc = no identity issued

or on the agent/server side:

level=error msg="Failed to authorize workload" subsystem_name=api error="no identity issued"

Step 5: Client and server perform mTLS using SVIDs

payments-api opens a TLS connection to ledger. The client presents its X.509-SVID. The server validates:

1. certificate chain to a trusted SPIFFE bundle
2. certificate time validity
3. SAN URI contains the expected SPIFFE ID format
4. authz policy allows that SPIFFE ID

If you are using a SPIFFE-aware TLS library, you verify the peer ID directly rather than hand-parsing SANs. That is the important distinction: TLS proves possession of a private key; SPIFFE-aware verification binds that proof to a workload identity your policy understands.

Step 6: Cross-cluster trust is established

If both clusters share the same trust domain and bundle, verification succeeds directly.

If they use different trust domains, add federation. Example trust domains:

• spiffe://prod-eu.example.org
• spiffe://prod-us.example.org

Then ledger must trust the prod-eu bundle and explicitly allow spiffe://prod-eu.example.org/ns/payments/sa/payments-api.

A common failure here is “TLS succeeded in the transport library, but app authz denied the call” because the peer cert was valid but from the wrong trust domain.

Typical server-side log shape:

tls: peer certificate verified, but SPIFFE ID spiffe://prod-eu.example.org/ns/payments/sa/payments-api not allowed by policy

Another common failure is stale or missing bundle propagation:

x509: certificate signed by unknown authority

That error usually means the remote trust bundle was never federated into this cluster, or bundle rotation happened and one side did not reload.

When to use it (and when not to)

You probably do not need this if all of these are true:

• everything runs in one cluster
• your mesh already enforces mTLS and identity-based policy adequately
• you do not have non-mesh clients
• your current cert issuance is not a bottleneck or incident source

Trade-offs

Latency impact is usually small after startup because SVID fetches are local over a Unix socket and TLS session reuse helps. The real cost is operational, not per-request CPU. Expect more pager noise from certificate expiry windows, trust bundle reload bugs, and registration drift than from raw handshake time.

Lock-in is lower than many mesh-specific identity systems because SPIFFE is a spec, but the operational details still depend on your chosen implementation and how deeply your apps assume a specific library or sidecar model.

In practice

Example 1: Create a registration entry in SPIRE

spire-server entry create \
  -spiffeID spiffe://prod.example.org/ns/finance/sa/ledger \
  -selector k8s:ns:finance \
  -selector k8s:sa:ledger \
  -dns ledger.finance.svc.cluster.local \
  -ttl 3600

This issues an X.509-SVID for pods in namespace finance using service account ledger, with a DNS SAN and a 1-hour TTL. Gotcha: if your app authorizes by SPIFFE ID, the -dns value is only for TLS name compatibility; do not accidentally authorize on DNS name and lose the benefit of workload identity.

Example 2: Verify a peer SPIFFE ID in Go using the Workload API

package main

import (
    "context"
    "crypto/tls"
    "log"
    "net/http"
    "time"

    "github.com/spiffe/go-spiffe/v2/spiffetls/tlsconfig"
    "github.com/spiffe/go-spiffe/v2/spiffeid"
    "github.com/spiffe/go-spiffe/v2/workloadapi"
)

func main() {
    ctx := context.Background()

    source, err := workloadapi.NewX509Source(ctx,
        workloadapi.WithClientOptions(workloadapi.WithAddr("unix:///run/spire/sockets/agent.sock")),
    )
    if err != nil {
        log.Fatalf("x509 source: %v", err)
    }
    defer source.Close()

    expected, err := spiffeid.FromString("spiffe://prod.example.org/ns/finance/sa/ledger")
    if err != nil {
        log.Fatalf("parse spiffe id: %v", err)
    }

    tlsConfig := tlsconfig.MTLSClientConfig(source, source, tlsconfig.AuthorizeID(expected))
    tlsConfig.MinVersion = tls.VersionTLS13

    client := &http.Client{
        Timeout: 5 * time.Second,
        Transport: &http.Transport{TLSClientConfig: tlsConfig},
    }

    resp, err := client.Get("https://ledger.finance.svc.cluster.local:8443/healthz")
    if err != nil {
        log.Fatalf("request failed: %v", err)
    }
    defer resp.Body.Close()

    log.Printf("status=%s", resp.Status)
}

This fetches the local workload SVID from the agent socket and only accepts a server whose SPIFFE ID is exactly spiffe://prod.example.org/ns/finance/sa/ledger. Gotcha: exact-ID authorization is intentionally strict; if you rotate to a new service account or trust domain, clients will hard-fail until you update policy.

Example 3: Inspect certificate SAN URIs with OpenSSL during debugging

openssl s_client -connect ledger.finance.svc.cluster.local:8443 -showcerts </dev/null 2>/dev/null \
| openssl x509 -noout -text | sed -n '/Subject Alternative Name/,+2p'

Typical good output shape:

X509v3 Subject Alternative Name:
    URI:spiffe://prod.example.org/ns/finance/sa/ledger, DNS:ledger.finance.svc.cluster.local

This is a quick sanity check that the server is actually presenting a SPIFFE URI SAN. Gotcha: OpenSSL can show you the cert, but it does not understand your SPIFFE authorization policy; a valid-looking SAN does not mean your app is verifying it.

⚠️ Changing trust bundles or trust domains can cause immediate cross-cluster outages if one side reloads before the other. Roll bundle changes in a staged overlap window where both old and new roots are trusted, and test with a single canary caller before broad rollout.

Further reading

• SPIFFE ID Specification
• SPIFFE X.509-SVID Specification
• SPIRE Server Configuration Reference
• SPIRE Federation Walkthrough
• NIST SP 800-207 Zero Trust Architecture

This article was written by an AI system and published pending human review. Verify anything you intend to act on.