Secrets in containers: env vars vs mounted files vs runtime fetch

TL;DR — Containers usually get secrets in one of three ways: environment variables, mounted files, or fetching them at runtime from a secret store. For most apps, use mounted files for certificates/keys and runtime fetch for high-value or frequently rotated credentials; use environment variables only for low-friction cases where restart-on-rotation is acceptable. Reading time: ~7 min

What it is and where it sits

This is the problem of getting sensitive values into a running container without baking them into the image, source tree, or CI logs.

In practice, the secret is usually one of these:

• database password or DSN
• API token
• TLS private key / certificate chain
• cloud credentials
• signing key

The delivery mechanism sits between your secret source and your application process. The source might be a container orchestrator secret object, a host file, or a networked secret manager. The consumer is your app process, often via its config loader at startup, sometimes via a sidecar or agent.

What it replaces:

• hardcoded values in source code
• .env files copied into images
• CI substituting secrets into config files during build
• long-lived static credentials committed to deployment manifests

Architecture context matters because the delivery method changes your blast radius and rotation story:

• Environment variables: injected by the container runtime before process start.
• Mounted files: exposed as files inside the container filesystem, often read-only.
• Runtime fetch: app or sidecar calls a secret service over the network and keeps/refreshes the value in memory or on disk.

[Secret Store] ---- 1. read secret ----> [Orchestrator/Agent]
[Orchestrator/Agent] ---- 2a. inject env ----> [Container]
[Orchestrator/Agent] ---- 2b. mount file ---> [Container]
[Container] ---- 3. connect with secret ----> [Database/API]

The important architecture question is not "where can I put the password?" but "who can read it, how does it rotate, and what breaks when it changes?"

How it actually works

Walk one realistic example: a containerized API needs a Postgres password that rotates every 24 hours.

We'll compare the same app under the three delivery models.

Step 1: The app starts

Assume the app reads DB_HOST, DB_USER, DB_PASSWORD, and opens a connection pool on boot.

With environment variables, the runtime starts the process like this:

docker run --rm \
  -e DB_HOST=db.internal \
  -e DB_USER=app \
  -e DB_PASSWORD='s3cr3t' \
  my-api:2026.10

Inside the container, the process sees the values in environ. On Linux, that often means they are visible via /proc/<pid>/environ to processes with sufficient access in the same namespace/host.

With a mounted file, the runtime exposes a file:

docker run --rm \
  --mount type=bind,src=/run/secrets/db_password,dst=/run/secrets/db_password,readonly \
  -e DB_HOST=db.internal \
  -e DB_USER=app \
  my-api:2026.10

The app reads /run/secrets/db_password on startup.

With runtime fetch, the app starts with only enough identity to authenticate to the secret store: for example a workload identity token or short-lived bootstrap credential. Then it calls the secret API on boot, gets DB_PASSWORD, and opens the pool.

Step 2: Rotation happens

At 02:00, the database password changes.

• Environment variable: the running process does not get updated. You must restart the container with a new env var value.
• Mounted file: depends on the platform. Some orchestrators update the mounted content atomically; your app still has to re-read the file and reopen connections. If it only read once at startup, rotation is effectively restart-required.
• Runtime fetch: app or sidecar can refresh on TTL/lease expiry and reopen connections without a full container restart.

This is where most teams discover the real issue is not delivery but reload behavior.

Step 3: Failure modes you actually see

If the password rotated and the app kept the old one, Postgres clients usually fail like this:

psql: error: connection to server at "db.internal" (10.0.12.14), port 5432 failed: FATAL: password authentication failed for user "app"

If your app is using runtime fetch and cannot reach the secret service, startup often fails before the app binds its port. A realistic log shape:

2026-10-30T02:00:02Z ERROR secrets: GET https://secrets.example.internal/v1/db/app: dial tcp 10.0.8.21:443: i/o timeout
2026-10-30T02:00:02Z FATAL config: could not load DB_PASSWORD from secret provider
exit status 1

If the secret service endpoint is misconfigured behind an HTTP redirect, curl -I often tells you immediately:

curl -I https://secrets.example.internal/v1/db/app
HTTP/1.1 301 Moved Permanently
Location: http://secrets.example.internal/v1/db/app/
Content-Length: 0

That is a bad sign for a secret endpoint: redirecting from HTTPS to HTTP or changing path shape can break clients and leak metadata. Fix the endpoint or client base URL before debugging auth.

Step 4: What a robust implementation does

A production-grade path for this example is usually:

1. container gets a short-lived identity token from the platform
2. app or sidecar exchanges identity for secret access
3. secret fetched over TLS
4. app caches in memory with explicit expiry
5. on refresh, app drains and recreates DB pool
6. if refresh fails, app keeps serving with last known good secret until expiry policy says stop

That last step is a policy decision. Fail-open preserves availability but extends secret lifetime in memory. Fail-closed is stricter but can cause an outage from a transient secret-store incident.

When to use it (and when not to)

You probably don't need runtime fetch if:

• the secret changes once a quarter
• restarting the deployment is routine and safe
• the app cannot reload config anyway
• you do not have a reliable secret service and identity plumbing yet

You probably should not use environment variables if:

• the secret is a private key or certificate bundle
• you need independent rotation without process restart
• you run debugging tooling that dumps process env on crashes
• multiple libraries may log config on startup

Trade-offs

Environment variables

Benefit: simplest operationally; every runtime supports them.

Cost: rotation usually means restart; values are easy to leak through process inspection, crash reports, debug endpoints, and accidental logging.

Benefit: app code often needs zero changes.

Cost: bad fit for multiline secrets and large blobs; quoting/newline handling causes subtle breakage.

Mounted files

Benefit: good fit for keys, certs, and structured credentials; file permissions give you one more control point.

Cost: your app must read files and ideally watch/reload them; many apps still only read once on boot.

Benefit: can support atomic updates from the platform.

Cost: update semantics vary; some mounts are symlinks or projected volumes, and file watchers may miss the change if you watch the wrong inode.

Runtime fetch

Benefit: best for short-lived credentials, auditability, dynamic rotation, and reducing secret sprawl in manifests.

Cost: highest complexity: network dependency, auth bootstrap, caching, retries, TTL handling, and outage modes.

Benefit: strongest alignment with least privilege and ephemeral credentials.

Cost: vendor and platform lock-in risk is higher because identity-to-secret integrations differ a lot.

In practice

Example 1: Docker Compose with env var and mounted file

services:
  api:
    image: my-api:2026.10
    environment:
      DB_HOST: db.internal
      DB_USER: app
      DB_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password
secrets:
  db_password:
    file: ./secrets/db_password.txt

This starts the container with a file-backed secret and tells the app where to read it. Gotcha: many apps support *_FILE conventions only if the image entrypoint implements them; otherwise you must add code to read /run/secrets/db_password yourself.

To inspect what actually arrived in the container:

docker compose exec api sh -lc 'ls -l /run/secrets && wc -c /run/secrets/db_password && env | grep ^DB_'

Typical output shape:

total 4
-r--r--r--    1 root     root            24 Oct 30 02:00 db_password
24 /run/secrets/db_password
DB_HOST=db.internal
DB_USER=app
DB_PASSWORD_FILE=/run/secrets/db_password

Example 2: Kubernetes Secret as env vs volume

⚠️ Updating a Kubernetes Secret does not magically refresh your app config unless your app re-reads it or your deployment restarts. If you switch from env vars to mounted files on a live service, plan a rollout because existing pods will not change behavior mid-flight.

apiVersion: v1
kind: Secret
metadata:
  name: api-db
stringData:
  username: app
  password: s3cr3t
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: my-api:2026.10
          env:
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: api-db
                  key: username
          volumeMounts:
            - name: db-secret
              mountPath: /var/run/secrets/db
              readOnly: true
      volumes:
        - name: db-secret
          secret:
            secretName: api-db
            items:
              - key: password
                path: password

This injects DB_USER as an env var and mounts the password at /var/run/secrets/db/password. Gotcha: if you use subPath mounts for secret files, updates are not reflected the same way as normal secret volumes; avoid subPath for rotating secrets.

Useful diagnostics:

kubectl get secret api-db -o yaml
kubectl exec deploy/api -- sh -lc 'printenv DB_USER; ls -l /var/run/secrets/db; cat /var/run/secrets/db/password | wc -c'
kubectl rollout restart deploy/api
kubectl rollout status deploy/api

If the app is still using the old password after a secret update and restart, inspect whether your connection pool is initialized before config reload or whether a sidecar/entrypoint copied the file elsewhere.

Example 3: Runtime fetch in app code

export SECRET_URL='https://secrets.example.internal/v1/db/app'
export SECRET_TOKEN_FILE='/var/run/secrets/tokens/workload'
python app.py

import os, time, requests

def read_token(path):
    with open(path, "r", encoding="utf-8") as f:
        return f.read().strip()

def fetch_db_password():
    token = read_token(os.environ["SECRET_TOKEN_FILE"])
    r = requests.get(
        os.environ["SECRET_URL"],
        headers={"Authorization": f"Bearer {token}"},
        timeout=2,
    )
    r.raise_for_status()
    return r.json()["password"]

if __name__ == "__main__":
    pw = fetch_db_password()
    print(f"loaded password length={len(pw)}")
    time.sleep(3600)

This fetches a secret at startup using a token from a mounted file. Gotcha: do not log the response body on non-200 errors; many teams add print(r.text) during debugging and end up dumping secret metadata or even values into centralized logs.

Further reading

• Kubernetes Documentation: "Secrets"
• Docker Docs: "Manage sensitive data with Docker secrets"
• The Twelve-Factor App: "Config"
• OWASP Secrets Management Cheat Sheet
• PostgreSQL Documentation: "Client Authentication"

This article was written by an AI system and published pending human review. Verify anything you intend to act on.