Audit Evidence for Key Management: Rotation, Access, Separation of Duties

TL;DR — Auditors do not want to hear "we use a KMS"; they want evidence that keys rotate on a defined schedule, only the right identities can use or administer them, and no single person can both change and approve sensitive key operations. The fastest path is to map each key to three artifacts: rotation configuration, access policy plus usage logs, and a change/approval trail showing separation of duties. Reading time: ~7 min

What it is and where it sits

For audits, "key management evidence" is the proof that your encryption keys are governed, not just present. In practice that means three things:

• Rotation evidence: when the key was created, whether rotation is enabled, what the period is, and proof that a new key version was actually used.
• Access evidence: who can administer the key, who can use it for encrypt/decrypt/sign, and logs showing actual use.
• Separation-of-duties evidence: proof that the same human cannot unilaterally create, grant, use, and retire sensitive keys without review.

This sits below your app code and above your storage layer. Your app usually talks to a KMS, HSM-backed service, Vault transit engine, or a cloud provider’s key service. That service then wraps or unwraps data-encryption keys used by databases, object stores, message payloads, backups, or token-signing systems.

What it replaces:

• SSH-ing into servers to copy PEM files around.
• Long-lived keys committed to CI variables or .env files.
• Shared admin accounts with no attributable audit trail.

Where it lives in a typical flow:

[Developer] ---- 1. PR changes key policy ----> [Git/CI]
[Git/CI] ---- 2. apply IAM/KMS config ----> [KMS]
[App] ---- 3. encrypt/decrypt via role ----> [KMS]
[KMS] ---- 4. use key version N ----> [Encrypted Data Store]
[Audit Logs] <---- 5. admin + usage events ---- [KMS]
[Auditor] ---- 6. review config + logs + approvals ----> [Evidence Set]

Architecturally, the important distinction is between:

• Key administrators: can create keys, change policies, schedule deletion, enable rotation.
• Key users: can encrypt/decrypt/sign/verify with a key but cannot change its policy.
• Approvers/reviewers: approve policy or lifecycle changes in Git, ticketing, or change-management tooling.

If those roles collapse into one IAM principal or one team with direct console access and no review trail, you have an audit problem even if the crypto is technically fine.

How it actually works

Walk one realistic example: a service encrypts customer documents with envelope encryption using a cloud KMS-backed symmetric key. You need evidence for a SOC 2 or ISO 27001 audit.

Step 1: Define the key and its owners in code

You create a customer-data key with aliases/tags that tie it to a system, owner, and classification. The exact tooling varies, but the evidence pattern is the same: infrastructure-as-code plus version control history.

Example evidence sources:

• Terraform file creating the key and alias.
• Git commit showing who proposed it.
• Pull request approval from someone outside the implementer’s role.

The auditor is looking for "this key exists for this purpose" and "someone reviewed the change."

Step 2: Separate admin permissions from usage permissions

Your app runtime role gets cryptographic use only. Your platform/security admin role gets policy management only.

Concretely, if you dump the effective policy, you should be able to point at statements equivalent to:

• app role: Encrypt, Decrypt, maybe GenerateDataKey
• security/platform role: Create*, Describe*, EnableKeyRotation, Put*Policy, ScheduleKeyDeletion
• no human developer role with both sets in production

If you are using a Unix-style secret store or Vault-like system, the equivalent is separate paths/capabilities. For example, app token can update on transit/encrypt/customer-data and transit/decrypt/customer-data; admin token can manage key config but is not mounted in app runtime.

Step 3: Turn on rotation and verify a new version exists

Rotation settings alone are not enough evidence. Auditors often ask for proof that rotation is configured and has happened.

Typical command shapes:

aws kms get-key-rotation-status --key-id alias/customer-data-prod

Typical output shape:

{
  "KeyRotationEnabled": true,
  "RotationPeriodInDays": 365
}

Then inspect metadata and usage timestamps:

aws kms describe-key --key-id alias/customer-data-prod

Output shape to look for:

{
  "KeyMetadata": {
    "KeyId": "1234abcd-...",
    "Description": "Customer document envelope key",
    "Enabled": true,
    "KeyManager": "CUSTOMER",
    "KeyUsage": "ENCRYPT_DECRYPT",
    "CreationDate": "2025-01-10T12:15:00+00:00",
    "MultiRegion": false,
    "Arn": "arn:aws:kms:...:key/1234abcd-..."
  }
}

For systems that expose key versions directly, capture the version list before and after the rotation window. In Vault transit, for example, latest_version, min_decryption_version, and keys metadata are the evidence, not just a checkbox.

Step 4: Capture usage logs tied to identities

Now prove the app used the key and that the calls are attributable.

For cloud KMS, pull audit events from your provider’s audit log service. You want event records for Encrypt, Decrypt, GenerateDataKey, DisableKey, PutKeyPolicy, ScheduleKeyDeletion, and rotation-related events. For self-hosted systems, use audit device logs or API gateway logs.

A realistic filter pattern is:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=alias/customer-data-prod \
  --max-results 20

Output shape:

{
  "Events": [
    {
      "EventName": "Decrypt",
      "Username": "arn:aws:sts::111122223333:assumed-role/app-prod-role/i-0abc...",
      "EventTime": "2026-10-20T09:14:33Z"
    },
    {
      "EventName": "PutKeyPolicy",
      "Username": "arn:aws:sts::111122223333:assumed-role/security-admin/jane",
      "EventTime": "2026-10-01T16:02:11Z"
    }
  ]
}

That is what lets you say: the app used the key under its runtime role; the admin changed policy under a different role.

Step 5: Prove separation of duties with a change trail

This is the part teams miss. IAM policy separation is necessary but not sufficient if one person can assume both roles or merge their own changes.

Useful evidence bundle:

• PR showing the key policy or IaC change.
• Required reviewer from security/platform team.
• CI apply logs showing the deployment identity, not a human laptop session.
• IAM role assignment docs or exports showing app role and admin role are distinct.
• If break-glass exists, evidence it is time-bound and logged.

A good audit answer is not "developers can’t do that." It is: "policy changes go through PR approval by security; CI applies using a deployment role; production app role cannot administer keys; all admin API calls are logged."

⚠️ Be careful with evidence collection commands that modify state. schedule-key-deletion, disable-key, vault write .../keys/.../rotate, or ad-hoc policy edits can cause outages or make old ciphertext undecryptable if combined with version retirement. For audit prep, prefer read-only commands like describe, get, list, and log queries.

When to use it (and when not to)

You probably don’t need a complex dedicated HSM program if:

• your provider-managed KMS already meets your regulatory and customer requirements,
• you do not need direct custody of hardware-backed keys,
• and your main gap is evidence, not cryptographic capability.

Trade-offs

• Managed KMS gives strong logs and policy controls — but costs money per key, per API call, and can add latency on hot paths.
• Automatic rotation reduces human failure — but can break brittle consumers that incorrectly pin key material or assume one static version.
• Strict admin/user separation improves audit posture — but increases operational friction when debugging production incidents.
• GitOps/IaC gives clean evidence — but only if console changes are blocked or at least detected; otherwise your source of truth drifts.
• Short-lived role-based access is better than shared secrets — but requires your CI, runtime, and human access model to be mature enough to issue and trace identities.
• Centralized key services reduce key sprawl — but create dependency on one control plane; outages or permission mistakes can become widespread incidents.

The practical balance for most software agencies is: managed KMS, app/runtime roles for use, separate admin roles for management, all changes through PR + CI, and periodic export of logs/config snapshots into an audit evidence folder.

In practice

Example 1: Read-only evidence collection for a cloud KMS key

KEY_ID="alias/customer-data-prod"
OUTDIR="evidence/$(date +%F)-customer-data-key"
mkdir -p "$OUTDIR"

aws kms describe-key --key-id "$KEY_ID" > "$OUTDIR/01-describe-key.json"
aws kms get-key-rotation-status --key-id "$KEY_ID" > "$OUTDIR/02-rotation-status.json"
aws kms list-resource-tags --key-id "$KEY_ID" > "$OUTDIR/03-tags.json"
aws kms get-key-policy --key-id "$KEY_ID" --policy-name default > "$OUTDIR/04-key-policy.json"
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue="$KEY_ID" \
  --max-results 50 > "$OUTDIR/05-cloudtrail-events.json"

This creates a minimal evidence pack you can attach to an audit ticket. Gotcha: lookup-events is shallow and time-bounded in many environments; for a real audit window, query your long-term log store instead of relying on the default event history.

Example 2: Vault transit key metadata and audit evidence

export VAULT_ADDR="https://vault.internal:8200"
export VAULT_TOKEN="$(cat ~/.vault-token)"

vault read -format=json transit/keys/customer-data > evidence/vault-key.json
vault audit list -format=json > evidence/vault-audits.json
vault token lookup -format=json > evidence/current-token.json

This captures the transit key metadata, enabled audit devices, and the current token identity used to read evidence. Gotcha: vault token lookup proves who you are now, not who changed the key last month; for that you need persisted audit device logs shipped to a log backend.

Example 3: IAM policy split between app use and key admin

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AppCanUseKey",
      "Effect": "Allow",
      "Action": ["kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey", "kms:DescribeKey"],
      "Resource": "arn:aws:kms:REGION:ACCOUNT:key/1234abcd-..."
    },
    {
      "Sid": "AppCannotAdministerKey",
      "Effect": "Deny",
      "Action": ["kms:PutKeyPolicy", "kms:ScheduleKeyDeletion", "kms:DisableKey", "kms:EnableKeyRotation"],
      "Resource": "*"
    }
  ]
}

Attach this shape to the app runtime role, with a separate admin policy for security/platform staff. Gotcha: explicit deny is powerful but broad; test it in staging because wildcard denies can block unrelated automation if you reuse roles carelessly.

Further reading

• NIST SP 800-57 Part 1: Key Management
• OWASP Key Management Cheat Sheet
• AWS KMS Developer Guide: Rotating keys
• HashiCorp Vault Documentation: Transit Secrets Engine and Audit Devices
• SOC 2 Trust Services Criteria, CC6 and CC7

This article was written by an AI system and published pending human review. Verify anything you intend to act on.