Break-glass vault access design and how to prove it was not used

TL;DR — A good break-glass procedure for a privileged access vault is a separate, tightly-scoped emergency path that works when your normal identity or control plane is down, but leaves tamper-evident evidence if it is used. The single biggest design mistake is reusing the same auth, network path, or logging backend as the primary vault path; if those fail together, your break-glass path is fiction, and if the logs are mutable, you cannot prove non-use. Reading time: ~7 min

What it is and where it sits

A break-glass procedure is the emergency route to retrieve or activate privileged credentials when your normal access path is unavailable or too slow for incident response. In a vault context, it usually exists because the primary path depends on things that fail in real incidents: SSO, MFA push, the control plane, a service mesh, a bastion, or the vault itself.

Architecturally, it sits beside the primary vault workflow, not inside it. If your normal flow is "engineer authenticates to IdP → gets short-lived token → reads secret from vault → connects to prod", the break-glass path should avoid at least one major shared dependency and have its own audit trail.

What it replaces during an emergency:

• Normal human access via SSO/OIDC/SAML into the vault UI or CLI.
• Standard approval workflow in ticketing/chat if those systems are impaired.
• Routine just-in-time elevation.

What talks to it:

• A small set of authorized humans, ideally requiring 2-person control.
• An offline or separately-administered secret store, hardware token, escrowed encrypted blob, or cloud KMS object with separate IAM.
• A write-once or append-only audit destination.
• Your incident command process.

Typical placement in the flow:

Normal path:
Engineer -> IdP/SSO -> Vault auth backend -> Vault policy -> Secret lease -> Target system

Break-glass path:
Engineer A + Engineer B -> Separate auth/material -> Emergency package/key release -> Target system
                                      |
                                      +-> Independent audit log / immutable storage

The key context: this is not a backup admin password taped under the keyboard. It is a controlled recovery mechanism with three properties:

1. It works when the primary path is broken.
2. It is narrow enough that using it is exceptional and obvious.
3. It leaves evidence strong enough that you can later say either "it was used at 03:14 UTC by these people" or "it was not used during the incident window."

How it actually works

Use one realistic model: an application team normally gets database admin credentials from a vault using OIDC login. Break-glass exists for the case where the IdP is down and a production incident requires direct Postgres access.

Design:

• The normal DB admin password is not stored as a standing secret in the app repo, CI, or wiki.
• A separate emergency credential exists, but it is disabled by default or sealed in encrypted form.
• Releasing it requires two custodians, each holding one decryption key share or one approval step in a separate IAM domain.
• Every release attempt writes to an append-only log sink outside the vault's own logging path.
• The target system also logs successful and failed authentication attempts for that emergency principal.

End-to-end example

Scenario: your IdP outage blocks vault login. Production Postgres is unhealthy. You need emergency admin access.

Step 1: Confirm the primary path is actually unavailable.

vault login -method=oidc role=prod-admin

Typical failure shape:

Error authenticating: error performing token check: Put "https://idp.example.com/oauth/token": dial tcp 203.0.113.10:443: i/o timeout

That error matters. If the vault itself is healthy but OIDC is dead, a break-glass path that still depends on OIDC is useless.

Step 2: Open an incident and record the intent to invoke break-glass. This is process, but do it with a literal command if your incident tooling supports API calls.

curl -sS -X POST https://incident-api.example.net/incidents \
  -H 'Content-Type: application/json' \
  -d '{"service":"payments-db","severity":"sev1","action":"request-break-glass","reason":"IdP outage blocks vault access; DB admin needed"}'

Step 3: Two custodians retrieve an encrypted emergency package from immutable object storage and decrypt it together on a hardened workstation.

For example, the package contains a one-time password envelope or instructions to enable a disabled DB role. The package itself is encrypted with age or GPG to two recipients, or split with Shamir secret sharing. The important part is not the brand; it is that one person cannot silently do it alone.

aws s3 cp s3://org-breakglass/prod/postgres-admin.age ./postgres-admin.age
age -d -i custodian-a.key -i custodian-b.key -o ./postgres-admin.txt ./postgres-admin.age

If one key is missing, the failure should be obvious:

age: error: no identity matched any of the recipients

Step 4: Use the released material against the target system. If the emergency account is disabled-by-default, enable it first from a separate admin plane; if it is a password envelope, use it directly.

PGPASSWORD="$(cat ./postgres-admin.txt)" psql "host=db-prod.internal port=5432 user=breakglass_admin dbname=postgres sslmode=require"

Step 5: The target system logs the auth. In Postgres, configure log_connections=on, log_disconnections=on, and include %u@%d %r in log_line_prefix so the break-glass role is visible.

Expected log shape:

2026-08-10 03:14:22 UTC [18422] breakglass_admin@postgres 198.51.100.24 LOG:  connection authorized: user=breakglass_admin database=postgres application_name=psql

Step 6: Rotate or disable immediately after use.

psql "host=db-prod.internal user=security_admin dbname=postgres sslmode=require" -c "ALTER ROLE breakglass_admin NOLOGIN;"

Step 7: Later, prove it was not used during an incident window by checking three independent evidence sources:

• Release log: no decrypt/retrieval event for the emergency package.
• Target auth log: no login by the emergency principal.
• Network or session log: no connection from the hardened workstation or break-glass bastion to the target.

This is the crucial point: you do not prove a negative from one system. You prove non-use by absence across independent, tamper-resistant logs with synchronized time.

When to use it (and when not to)

You probably do not need this if all of the following are true:

• Production can be safely recovered without human privileged access.
• Your normal vault path has no shared dependency on a brittle external IdP or control plane.
• You can tolerate waiting for the primary system to recover.
• You are not under compliance or customer obligations to demonstrate emergency access controls.

Trade-offs

Every benefit costs something.

• Separate emergency path reduces incident blast radius.
  • Cost: more moving parts, more runbooks, more things to test quarterly.
• Dual control reduces insider risk.
  • Cost: slower activation at 02:00, scheduling pain, and risk of both custodians being unavailable.
• Immutable audit logs improve proof of non-use.
  • Cost: extra storage, retention design, and operational work to export logs out of the primary stack.
• Disabled-by-default emergency accounts reduce standing exposure.
  • Cost: activation itself becomes a critical operation that must also be audited and tested.
• Offline escrow avoids dependency on the vault.
  • Cost: rotation is harder; stale escrow is a common failure mode.
• Narrow-scoped emergency credentials reduce damage.
  • Cost: in a real incident they may be too narrow, forcing a second emergency process.

Lock-in note: avoid designs that only one vendor can verify. If your proof of non-use depends entirely on a proprietary audit UI, export logs in a standard format to object storage or syslog-compatible collectors. You want evidence you can query with ordinary tools.

In practice

Example 1: Postgres logging that makes break-glass use visible

# postgresql.conf
log_connections = on
log_disconnections = on
log_line_prefix = '%m [%p] %u@%d %r '
log_statement = 'ddl'

This makes successful break-glass logins visible by username, database, and client IP. Gotcha: log_connections only shows successful auth; failed attempts may require log_min_messages and host-based auth logging via pg_hba.conf plus system logs.

To verify after an incident window:

grep 'breakglass_admin@' /var/log/postgresql/postgresql-*.log

If there was no use, output is empty and grep exits 1.

$ grep 'breakglass_admin@' /var/log/postgresql/postgresql-*.log
$ echo $?
1

That exit code is useful in automation, but do not treat it as proof by itself; pair it with independent evidence.

Example 2: Append-only release log written to object storage

ts="$(date -u +%FT%TZ)"
incident="INC-4821"
actor="custodian-a"
printf '{"ts":"%s","incident":"%s","actor":"%s","action":"retrieve-breakglass-package","object":"prod/postgres-admin.age"}
' "$ts" "$incident" "$actor" | tee /tmp/breakglass-event.json
aws s3 cp /tmp/breakglass-event.json s3://org-breakglass-audit/events/${ts}-${incident}-${actor}.json \
  --sse AES256 \
  --expected-size $(wc -c < /tmp/breakglass-event.json)

This writes a simple event record to separate storage before decryption. Gotcha: if the same IAM role can both write and delete these objects, your audit is not tamper-evident; use bucket versioning, object lock/WORM where available, and a separate admin boundary.

To query whether any release happened during a window:

aws s3api list-objects-v2 \
  --bucket org-breakglass-audit \
  --prefix events/2026-08-10T03: \
  --query 'Contents[].Key'

Typical no-event output:

[]

Example 3: A disabled-by-default emergency role in Postgres

⚠️ Enabling a superuser-capable role can create immediate production risk. Run this only from a controlled admin session, record the incident ID first, and disable it again as soon as the task is complete.

CREATE ROLE breakglass_admin LOGIN PASSWORD 'REPLACE-ME' NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT;
GRANT pg_signal_backend TO breakglass_admin;
GRANT CONNECT ON DATABASE postgres TO breakglass_admin;
ALTER ROLE breakglass_admin NOLOGIN;

Then activate only during an incident:

ALTER ROLE breakglass_admin LOGIN PASSWORD 'new-random-rotated-secret';

This pattern avoids a permanently usable account. Gotcha: if your emergency tasks really require superuser, be explicit about that and log every command through session capture or pgaudit; pretending a limited role is enough when it is not just pushes people toward undocumented workarounds.

Further reading

• NIST SP 800-63 Digital Identity Guidelines
• NIST SP 800-57 Key Management Guidelines
• PostgreSQL Documentation: Error Reporting and Logging
• The "Logging" and "Authentication" sections of your cloud provider's object storage docs
• Google SRE Book, the chapter on Emergency Response

This article was written by an AI system and published pending human review. Verify anything you intend to act on.