Test Passes Locally but Fails in CI: Diagnose Env, Order, Time, Concurrency

TL;DR — When a test passes on your laptop and fails in CI, the usual culprit is not the assertion itself but hidden assumptions: different env vars, a leaked dependency on test order/shared state, timezone/clock behavior, or parallel execution. Start by rerunning the exact failing test in a CI-like container with the CI env file and with parallelism disabled; that usually tells you which branch to follow. Reading time: ~6 min

The scenario

It is Tuesday at 3:40 PM. You push a small PR that only "touches a serializer," but CI lights up red on one test file that has been green for weeks. You rerun the job and it fails in a different test, while the same branch passes instantly on your laptop. The logs show no compile errors, just assertions that look impossible given what you just ran locally, and now the merge queue is backing up.

Symptoms

• A test passes locally with the normal command, but fails in CI with the same nominal suite target.
• Failures move around between runs, or only happen when the whole suite runs.
• Typical messages:

AssertionError: expected 3 to equal 2

Expected: "2026-03-31"
Received: "2026-03-30"

E   KeyError: 'API_BASE_URL'
E   AssertionError: assert None == 'https://example.test'

Error: connect ECONNREFUSED 127.0.0.1:5432

psycopg.errors.UniqueViolation: duplicate key value violates unique constraint "users_email_key"

FAIL src/foo.test.ts
  ● should sort newest first

    expect(received).toEqual(expected) // deep equality

    - Expected  - 1
    + Received  + 1
      Array [
    -   "b",
        "a",
    +   "b",
      ]

--- FAIL: TestExpiryAtMidnight (0.00s)
    expiry_test.go:41: got 2026-10-28 23:00:00 +0000 UTC want 2026-10-29 00:00:00 +0100

• CI logs mention worker counts or sharding:

Running tests with 8 workers

• Re-running a single test in CI passes, but the full suite fails.

Likely causes

| Test ordering / shared state leakage between tests | Very common | ```bash
pytest -q --random-order --random-order-bucket=global -x
``` |
| Clock / timezone / locale assumptions | Common | ```bash
TZ=UTC date && locale
``` |
| Concurrency / race condition / parallel test interference | Common | ```bash
pytest -q -n 0 -x
``` |
| Non-deterministic data generation or unordered queries/collections | Moderate | ```bash
git grep -nE "ORDER BY|sort\(|map iteration|rand\(|Math.random|uuid|time.Now\(|Date.now\(" test src
``` |

## Step-by-step diagnosis
1. **Capture the exact failing test and rerun only that test locally.**
```bash
# Examples; use your framework's exact selector
pytest -q tests/path/test_file.py::test_name -vv
npm test -- --runInBand src/foo.test.ts -t "should sort newest first"
go test ./... -run '^TestExpiryAtMidnight$' -count=1 -v

If the single test fails locally too, this is not CI-specific; stop here and debug the test normally. If it passes locally but fails in CI, continue to step 2.

1. Run in a CI-like environment, not your host shell.

# Minimal generic pattern: use the same image your CI job uses
docker run --rm -it -v "$PWD:/work" -w /work --env-file .ci.env node:22 bash -lc 'node -v && npm ci && npm test -- --runInBand src/foo.test.ts -t "should sort newest first"'

This is your problem if the failure reproduces in the container but not on your host. Jump to Fixes → Environment drift.

1. Disable parallelism and retries.

pytest -q -n 0 --maxfail=1 -vv
npm test -- --runInBand --verbose
go test ./... -p 1 -parallel 1 -count=1

If the suite passes only when parallelism is disabled, you have a race/shared-resource problem. Jump to Fixes → Concurrency / race condition.

1. Change test order or isolate the failing file with neighbors.

# Python
pytest -q --random-order --random-order-bucket=global -x
# Jest: run suspect files together serially
npx jest --runInBand src/a.test.ts src/b.test.ts src/foo.test.ts

This is your problem if a different order causes failures, or if the failing test only breaks when another file runs first. Jump to Fixes → Test ordering / shared state leakage.

1. Force UTC and a stable locale.

TZ=UTC LC_ALL=C.UTF-8 LANG=C.UTF-8 pytest -q tests/path/test_file.py::test_name -vv
TZ=UTC LC_ALL=C.UTF-8 LANG=C.UTF-8 npm test -- --runInBand src/foo.test.ts -t "date"

If the failure disappears under forced timezone/locale, or the expected date shifts by hours/days, jump to Fixes → Clock / timezone / locale assumptions.

1. Look for nondeterminism in ordering and generated values.

git grep -nE "ORDER BY|sort\(|Math.random|rand\(|Date.now\(|time.Now\(|uuid|map\[" -- .

This is your problem if assertions depend on insertion order from a DB query without ORDER BY, iteration order of maps/objects, random IDs, or current time. Usually fix under Ordering or Clock depending on what you find.

1. Compare CI env and service availability.

printenv | sort | sed 's/=.*$/=REDACTED/'
getent hosts postgres || true
curl -fsS http://127.0.0.1:3000/health || true

If variables are absent, hosts do not resolve, or dependent services are not listening, jump to Fixes → Environment drift.

Fixes

Environment drift: missing/different env vars, OS, locale, dependency or service versions

Pin the runtime and install path exactly as CI does.

# Node example
node -v
cat .nvmrc
npm ci
npm ls --depth=0

# Python example
python --version
pip install -r requirements.txt
pip freeze | sort

# Capture env without secrets
printenv | sort | sed 's/=.*$/=REDACTED/' > local-env-snapshot.txt

Create a committed CI env template and load it locally:

cp .env.example .ci.env
# edit .ci.env to include test-safe values only
set -a; . ./.ci.env; set +a

If CI depends on services, start them explicitly:

docker compose up -d postgres redis
docker compose ps

If locale matters, pin it in CI and local scripts:

export TZ=UTC
export LANG=C.UTF-8
export LC_ALL=C.UTF-8

Verify it worked:

docker run --rm -v "$PWD:/work" -w /work --env-file .ci.env node:22 bash -lc 'npm ci && npm test -- --runInBand'

Test ordering / shared state leakage between tests

Reset all mutable state between tests: database rows, files, caches, singleton objects, global mocks, process env changes.

For DB-backed tests, wrap each test in a transaction and roll it back, or truncate deterministically:

psql "$TEST_DATABASE_URL" -c "TRUNCATE TABLE users, orders, sessions RESTART IDENTITY CASCADE;"

Jest example for mock cleanup:

beforeEach(() => {
  jest.restoreAllMocks();
  jest.clearAllMocks();
  process.env = { ...originalEnv };
});

Pytest example for temp dirs and monkeypatch:

def test_thing(tmp_path, monkeypatch):
    monkeypatch.setenv("API_BASE_URL", "https://example.test")

If the failure is due to unordered DB results, make the query explicit:

SELECT id, created_at FROM jobs ORDER BY created_at DESC, id DESC;

Verify it worked:

pytest -q --random-order --random-order-bucket=global

Clock / timezone / locale assumptions

Stop asserting on wall-clock "now" directly. Inject a clock or freeze time in tests.

Node example:

jest.useFakeTimers().setSystemTime(new Date('2026-10-29T00:00:00Z'));

Python example:

from freezegun import freeze_time

@freeze_time("2026-10-29T00:00:00Z")
def test_expiry():
    ...

Go example: pass a clock function instead of calling time.Now() inside logic.

type Clock func() time.Time
func Expiry(now Clock) time.Time { return now().UTC().Truncate(24 * time.Hour) }

Normalize timezone handling in app code:

export TZ=UTC

Use timezone-aware values, not naive local timestamps. For formatting-sensitive tests, pin locale too:

export LANG=C.UTF-8
export LC_ALL=C.UTF-8

Verify it worked:

TZ=UTC LC_ALL=C.UTF-8 LANG=C.UTF-8 pytest -q tests/path/test_time.py -vv

Concurrency / race condition / parallel test interference

If tests share ports, files, DB rows, queues, or cache keys, parallel workers will collide. Give each worker isolated resources.

Use unique temp paths and ports:

export TEST_TMPDIR="$(mktemp -d)"
export PORT=0

For Postgres, isolate by schema or database per worker:

createdb app_test_1
createdb app_test_2
# map worker index to DB name in test bootstrap

For Redis/cache keys, namespace by worker:

export CACHE_NAMESPACE="test-${CI_NODE_INDEX:-0}-${PYTEST_XDIST_WORKER:-gw0}"

If the code itself races, run the race detector where available.

go test ./... -race -count=1

⚠️ Creating or dropping test databases can destroy data if you point at the wrong server. Run echo "$TEST_DATABASE_URL" and confirm it is a test instance before executing any create/drop command. If the framework supports it, keep parallelism off until isolation is fixed:

pytest -q -n 0
npx jest --runInBand
go test ./... -p 1 -parallel 1

Verify it worked:

for i in 1 2 3; do pytest -q -n auto || break; done

Non-deterministic data generation or unordered queries/collections

Replace random/current values with fixed fixtures or seeded generators.

Seed randomness explicitly:

export TEST_SEED=12345

Node example:

const seed = Number(process.env.TEST_SEED || 12345);

Avoid assertions on map/object iteration order unless the language guarantees it. Sort before asserting:

expect(items.map(x => x.id).sort()).toEqual(['a', 'b']);

For SQL, always specify order if order matters:

SELECT * FROM events ORDER BY occurred_at ASC, id ASC;

Verify it worked:

for i in 1 2 3 4 5; do TEST_SEED=$i npm test -- --runInBand || break; done

Prevention

• Pin runtime versions in both local dev and CI.

{
  "engines": { "node": "22.x" }
}

# .tool-versions or equivalent
nodejs 22.12.0
python 3.12.6

• Add a CI job that runs the suite in a different order and with a fixed timezone.

TZ=UTC LANG=C.UTF-8 LC_ALL=C.UTF-8 pytest -q --random-order --random-order-bucket=global

• Add a serial smoke run for flaky detection alongside the normal parallel run.

pytest -q -n auto && pytest -q -n 0 --maxfail=1

• Fail fast on missing env vars at process start instead of deep inside tests.

: "${API_BASE_URL:?missing API_BASE_URL}"
: "${TEST_DATABASE_URL:?missing TEST_DATABASE_URL}"

• Add race/concurrency checks where your stack supports them.

go test ./... -race -count=1

• Snapshot the CI environment for debugging, redacting values.

printenv | sort | sed 's/=.*$/=REDACTED/' > ci-env-snapshot.txt
uname -a >> ci-env-snapshot.txt
locale >> ci-env-snapshot.txt
date >> ci-env-snapshot.txt

That snapshot turns "works on my machine" into a diff you can act on.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.