Control test failed: evidence was collected from the wrong environment

TL;DR — Your control test usually fails here because the evidence collector hit a non-production URL, used the wrong cloud account/cluster, or authenticated with credentials scoped to the wrong tenant/project. Start by proving the target with one immutable identifier from the evidence itself: hostname, account ID, cluster name, database name, or environment banner; then compare it to the approved production inventory and fix the source config, not the screenshot. Reading time: ~6 min

The scenario

It is Tuesday afternoon, you are closing out audit evidence, and a control test comes back failed with a note like "evidence appears to be from staging." You know the feature exists in production, the deploy went out last week, and the screenshot in the ticket looks right at a glance. Then you notice the hostname in the browser chrome is app-staging.example.com, or the CLI output shows the wrong AWS account ID, or the database dump header says db_dev. Now you need to prove exactly where the evidence came from, correct it, and avoid re-running half the control set.

Symptoms

• Auditor or reviewer comment says one of:

  Evidence does not demonstrate production configuration.
  Screenshot appears to be from a non-production environment.
  Account/tenant identifier in evidence does not match approved production inventory.
  Control test failed: evidence source could not be tied to production.
  

• Screenshot or exported report contains obvious non-prod markers:

  staging
  dev
  sandbox
  qa
  test
  

• HTTP headers or redirects point to a non-prod host:

  $ curl -I https://app.example.com/admin
  HTTP/2 302
  location: https://app-staging.example.com/login
  server: nginx
  

• Cloud CLI output shows the wrong account/project/subscription:

  $ aws sts get-caller-identity
  {
    "UserId": "AIDAXXXXX",
    "Account": "111111111111",
    "Arn": "arn:aws:iam::111111111111:user/alice"
  }
  

• Kubernetes context or namespace is non-prod:

  $ kubectl config current-context
  gke-myco-staging-us-central1-app
  

• Database/session evidence references the wrong DB or schema:

  current_database
  ----------------
  app_staging
  

• App UI shows a banner or footer label inconsistent with production:

  Environment: Staging
  Build: 2026.10.27-rc1
  

Likely causes

Step-by-step diagnosis

1. Check the final URL actually reached by the evidence collector.

   curl -I -L -o /dev/null -w 'effective=%{url_effective}\ncode=%{http_code}\n' https://app.example.com/admin
   

   This is your problem if effective= ends with a staging/dev/sandbox hostname or path. Example:

   effective=https://app-staging.example.com/login
   code=200
   

   Jump to Fixes → Wrong URL/host or redirect.

2. Verify the environment identifier embedded in the evidence itself. For screenshots/PDFs, inspect metadata and visible strings:

   strings evidence.pdf | grep -Ei 'staging|sandbox|dev|qa|prod|production|example.com' | head -20
   exiftool evidence.png | grep -E 'Create Date|Modify Date|Software|URL'
   

   This is your problem if the file contains a non-prod hostname, banner, or old timestamp inconsistent with the collection window. Jump to Fixes → Stale cached or previously exported evidence attached.

3. Confirm the cloud account/project/tenant used during collection. AWS:

   aws sts get-caller-identity
   aws configure list
   

   Azure:

   az account show --output json
   

   GCP:

   gcloud config list --format='text(core.account,core.project)'
   

   This is your problem if the account/project ID does not match your approved production inventory. Jump to Fixes → Wrong cloud account/project/tenant.

4. Confirm Kubernetes context and namespace if the evidence came from cluster resources.

   kubectl config current-context
   kubectl config view --minify --output 'jsonpath={..namespace}{"\n"}'
   kubectl cluster-info
   

   This is your problem if the context name, API server, or namespace is a non-prod target. Jump to Fixes → Wrong Kubernetes context or namespace.

5. Inspect the environment variables or config file used by the evidence script/job.

   env | grep -E 'ENV|ENVIRONMENT|BASE_URL|ACCOUNT|PROJECT|CLUSTER|DATABASE_URL' | sort
   grep -RInE 'BASE_URL|ENVIRONMENT|DATABASE_URL|cluster|project|account' .
   

   This is your problem if BASE_URL, ENVIRONMENT, DATABASE_URL, or similar resolves to staging/dev, or if the repo has a default config pointing there. Jump to Fixes → Wrong environment variables or config in script/CI.

6. Verify the database endpoint if the evidence was SQL output.

   psql "$DATABASE_URL" -c 'select current_database(), current_user, inet_server_addr();'
   

   This is your problem if the DB name or server IP belongs to staging/dev, or if you are on a read replica that mirrors non-prod. Jump to Fixes → Wrong database connection string.

7. If all identifiers look correct, compare against the production inventory/source of truth. Use whatever your org treats as authoritative: CMDB, infra repo, Terraform outputs, or account registry. A mismatch here means the evidence may be valid but your inventory is stale; update the inventory first, then recollect evidence so the reviewer can tie it back cleanly.

[Evidence File] ---- 1. inspect host/account ----> [Observed Target]
      |
      | 2. compare
      v
[Approved Prod Inventory] ---- 3. mismatch? ----> [Wrong Env Config]
      |
      | 4. match
      v
[Recollect Fresh Evidence]

Fixes

Wrong URL/host or redirect

Update the evidence source to the canonical production URL, then fix redirects/bookmarks so the next run cannot drift.

Check redirect chain:

curl -I https://app.example.com/admin
curl -I https://app-staging.example.com/admin

Typical bad output:

HTTP/2 302
location: https://app-staging.example.com/login

If nginx is redirecting incorrectly, fix the server block:

server {
    listen 443 ssl http2;
    server_name app.example.com;

    location / {
        proxy_set_header Host $host;
        proxy_pass http://app_prod_upstream;
    }
}

Remove hard-coded staging URLs from scripts:

sed -i.bak 's#https://app-staging.example.com#https://app.example.com#g' scripts/collect-evidence.sh

If the issue is a stale bookmark or copied link, replace it in the runbook/automation source and re-run collection.

Verify it worked:

curl -I -L -o /dev/null -w '%{url_effective}\n' https://app.example.com/admin

Wrong cloud account/project/tenant

Switch to the approved production identity before collecting evidence.

AWS:

aws sso login --profile prod
AWS_PROFILE=prod aws sts get-caller-identity

If you use named profiles in scripts:

export AWS_PROFILE=prod

Azure:

az login
az account set --subscription "Production Subscription"
az account show --output table

GCP:

gcloud auth login
gcloud config set project my-prod-project
gcloud config list --format='text(core.account,core.project)'

Trade-off: switching global CLI context is fast but risky on a shared shell. Prefer per-command scoping like AWS_PROFILE=prod aws ... in automation.

Verify it worked:

aws sts get-caller-identity

Wrong Kubernetes context or namespace

Select the production cluster and namespace explicitly; do not rely on the current shell context.

kubectl config get-contexts
kubectl config use-context prod-cluster
kubectl config set-context --current --namespace=app-prod
kubectl config current-context
kubectl get ns

For scripts, pin both context and namespace on every command:

kubectl --context=prod-cluster --namespace=app-prod get deploy,po,svc

⚠️ Changing the current kubectl context affects subsequent commands in that shell. If you also deploy from the same terminal, use --context and --namespace flags instead of use-context.

Verify it worked:

kubectl --context=prod-cluster --namespace=app-prod get deploy

Wrong environment variables or config in script/CI

Set production values explicitly in the evidence job and fail hard if they are missing or non-prod.

Shell script example:

export ENVIRONMENT=production
export BASE_URL=https://app.example.com
export AWS_PROFILE=prod

case "$BASE_URL" in
  *staging*|*dev*|*sandbox*|*qa*) echo "Refusing non-prod BASE_URL=$BASE_URL"; exit 42 ;;
esac

CI config example:

steps:
  - name: collect-evidence
    env:
      ENVIRONMENT: production
      BASE_URL: https://app.example.com
      AWS_PROFILE: prod
    run: ./scripts/collect-evidence.sh

If your script reads .env, point it at the prod file explicitly:

ENV_FILE=.env.production ./scripts/collect-evidence.sh

Verify it worked:

env | grep -E 'ENVIRONMENT|BASE_URL|AWS_PROFILE'

Wrong database connection string

Replace the DSN with the production endpoint and annotate the evidence with DB identity.

export DATABASE_URL='postgres://evidence_user:***@prod-db.example.com:5432/app_prod?sslmode=require'
psql "$DATABASE_URL" -c 'select current_database(), current_user, inet_server_addr();'

If the DSN is in a config file:

grep -RIn 'DATABASE_URL' .
sed -i.bak 's/app_staging/app_prod/' .env.production

Edge case: if prod and staging share a replica host, inet_server_addr() alone may not distinguish them. Use current_database() and a known prod-only table/value.

Verify it worked:

psql "$DATABASE_URL" -c 'select current_database();'

Stale cached or previously exported evidence attached

Delete the old artifact, recollect, and include collection timestamp plus environment identifier in the file name.

stat evidence.pdf
rm -f evidence.pdf
./scripts/collect-evidence.sh
mv evidence.pdf evidence-production-$(date +%F-%H%M%S).pdf

If browser cache was involved, recollect in a fresh session:

google-chrome --incognito https://app.example.com

Or use a clean HTTP fetch instead of screenshots where possible:

curl -sS https://app.example.com/healthz -D headers.txt -o body.txt

Verify it worked:

stat evidence-production-*.pdf

Prevention

• Pin environment identifiers in evidence output names and content.

  OUT="control-7.2-production-$(date +%F-%H%M%S).txt"
  {
    echo "env=production"
    echo "url=https://app.example.com"
    aws sts get-caller-identity
  } > "$OUT"
  

• Add a preflight guard to evidence scripts that exits non-zero on non-prod targets.

  case "$BASE_URL $KUBE_CONTEXT $DATABASE_URL" in
    *staging*|*dev*|*sandbox*|*qa*) echo "non-prod target detected"; exit 42 ;;
  esac
  

• In CI, compare observed account/project/cluster against approved values stored in code.

  test "$(aws sts get-caller-identity --query Account --output text)" = "999999999999"
  test "$(kubectl config current-context)" = "prod-cluster"
  

• Store production inventory in version control and reference it during collection.

  {
    "production": {
      "base_url": "https://app.example.com",
      "aws_account": "999999999999",
      "kube_context": "prod-cluster",
      "database": "app_prod"
    }
  }
  

• Prefer machine-readable evidence over screenshots when possible: headers, CLI JSON, SQL output. It is easier to diff and harder to mislabel.

  curl -sS -D - https://app.example.com/login -o /dev/null
  aws iam get-account-summary --output json > account-summary-prod.json
  

• Add reviewer checks to the runbook template so every evidence item includes at least one immutable environment marker: hostname, account ID, cluster name, or DB name.

  - [ ] Hostname visible
  - [ ] Cloud account/project ID visible
  - [ ] Cluster/namespace or DB name visible
  - [ ] Collection timestamp visible
  

This article was written by an AI system and published pending human review. Verify anything you intend to act on.