Ransomware response in the first hours: isolate systems and protect backups

TL;DR — In the first hours of ransomware response, your job is not "restore fast"; it is to stop spread, preserve evidence, and determine whether identity systems, admin tooling, and backups are also compromised. The single most important move is immediate isolation at the network and credential layer before touching files or rebooting hosts. Reading time: ~7 min

What it is and where it sits

Ransomware response for engineering teams sits between incident response, infrastructure operations, identity/security administration, and disaster recovery. In practice, engineering owns the systems that can spread the blast radius: Kubernetes nodes, CI runners, bastion hosts, database servers, backup jobs, object storage credentials, and the automation that can either help contain the event or amplify it.

The architecture context matters because ransomware rarely stays "on one VM." It typically moves through one or more of these paths:

• reused SSH keys or cloud API keys
• domain admin / SSO sessions
• mounted network shares and backup repositories
• CI/CD secrets and deployment agents
• management planes: hypervisor, Kubernetes, MDM, RMM, backup console

What it replaces: normal ops priorities. During the first hours, you stop doing availability-first operations and switch to containment-first operations. That means no rolling restarts, no auto-healing that reattaches infected volumes, no "just restore from snapshot" until you know snapshots and credentials are trustworthy.

A typical flow looks like this:

[User Endpoint] ---- 1. stolen creds ----> [SSO/AD]
[SSO/AD] ---- 2. admin session ----> [VPN/Bastion]
[VPN/Bastion] ---- 3. SSH/API ----> [Prod Hosts]
[Prod Hosts] ---- 4. encrypt/share ----> [DB/File Shares]
[Prod Hosts] ---- 5. backup creds ----> [Backup Repo]
[IR Team]
|
v
[Network Controls] ---- 6. isolate ----> [Prod Hosts]
[IR Team] ---- 7. revoke tokens/keys ----> [SSO/Cloud IAM]
[IR Team] ---- 8. verify clean restore ----> [Backup Repo]

The key architectural question is: what still has trust? If your IdP, backup server, or cloud control plane is compromised, your response path changes immediately.

How it actually works

Here is one realistic end-to-end example: a Linux application fleet on VMs, Postgres on separate hosts, backups to object storage via a backup server, and engineers authenticate through SSO plus SSH certificates.

The trigger

Pager fires because app instances start returning 500s, disk write latency spikes, and a developer notices files renamed with an extension like .locked. On one host:

sudo find /srv/app -maxdepth 2 -type f | head

Output shape:

/srv/app/current/config.yml.locked
/srv/app/current/static/app.js.locked
/srv/app/current/README.locked

At the same time, Postgres logs show mass I/O failures because the underlying volume is saturated, not because Postgres itself is "the ransomware target":

sudo journalctl -u postgresql -n 20 --no-pager

postgres[1821]: could not write to file "base/16384/2619": No space left on device
postgres[1821]: checkpoint request failed
postgres[1821]: PANIC: could not fsync file "pg_wal/00000001000000AF000000C3": Input/output error

First decision: isolate before triage

⚠️ Isolation can cause immediate downtime. Do it anyway if you have active encryption, credential abuse, or lateral movement. Waiting to "collect a bit more info" is how one host becomes fifty.

The first actions are mechanical:

1. stop east-west and outbound traffic from suspected hosts
2. revoke active admin access paths
3. freeze automation that will propagate bad state

On a Linux host where you still have shell access, a crude but effective local isolation step is:

sudo iptables -P INPUT DROP
sudo iptables -P OUTPUT DROP
sudo iptables -P FORWARD DROP
sudo iptables -A INPUT -i lo -j ACCEPT
sudo iptables -A OUTPUT -o lo -j ACCEPT
sudo iptables -L -n -v

If SSH is your only remaining management path and you need to keep one IR jump host connected, add that rule first:

sudo iptables -A INPUT -p tcp -s 203.0.113.10 --dport 22 -j ACCEPT
sudo iptables -A OUTPUT -p tcp -d 203.0.113.10 --sport 22 -j ACCEPT

At the network edge, remove instances from load balancers or security groups in your provider dashboard or via CLI. For example, in a cloud environment, detach the instance from the app security group and attach a quarantine group that allows only your IR subnet.

Then revoke access that can re-enter the environment:

• disable compromised user accounts in your IdP/AD
• revoke cloud API keys and active sessions
• pause CI/CD runners and deployment bots
• disable backup jobs and replication jobs temporarily

Why pause backups? Because many backup systems will faithfully back up encrypted garbage or propagate deletions to replicas.

Second decision: preserve evidence or rebuild immediately

If the host is disposable stateless app infrastructure, preserve enough evidence to understand scope, then rebuild from known-good images. If it is a database, bastion, domain controller, or backup server, preserve more aggressively before changing state.

Minimal evidence collection on Linux before shutdown/rebuild:

date -Is
hostnamectl
who
last -a | head -20
ss -plant
ps auxwf --sort=-%cpu | head -40
sudo journalctl --since "-2 hours" --no-pager | tail -200
sudo tar -czf /var/tmp/ir-hostname-logs.tgz /var/log

If common tools start failing with Permission denied, Input/output error, or binaries appear replaced, stop trying to be clever on-host. Collect what you can from the hypervisor, cloud serial console, EDR, or centralized logs.

Third decision: are backups trustworthy?

This is the decision that determines whether you recover in hours or re-infect yourself.

Check three things:

1. immutability/retention: can recent restore points be deleted or altered by the same credentials that were compromised?
2. time window: do you have restore points from before the first known malicious activity, not just before encryption started?
3. restore path isolation: can you restore into a clean network/account without reconnecting to compromised identity and automation?

For Postgres, do not assume filesystem snapshots are application-consistent unless your backup process guarantees it. Verify the backup catalog and test a restore into an isolated host.

A quick object-store sanity check from a clean workstation can reveal whether backup artifacts are still present and whether credentials still work:

aws s3 ls s3://company-backups/postgres/base/ --summarize

Failure modes matter:

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

That may be good or bad. Good if you intentionally revoked credentials. Bad if the only credentials you have are gone and no break-glass path exists.

Or:

2026-10-30 02:10:11   104857600 base_2026-10-30.tar.gz
2026-10-30 03:10:11        1024 base_2026-10-30.tar.gz.README
Total Objects: 2
Total Size: 104858624

A tiny unexpected object count where you normally have many WAL/archive files is a red flag for deletion or lifecycle damage.

Recovery path

Restore only into a quarantine environment first. For the Postgres example:

• new VM in isolated subnet/account/project
• no trust to production SSO; use break-glass local admin or separate IAM role
• restore backup
• run integrity checks and application smoke tests against copied data
• only then promote clean data/app images back into production

If the backup server itself was reachable with domain admin or shared service credentials, assume backup metadata may be tampered with until proven otherwise.

When to use it (and when not to)

You probably don't need a full ransomware playbook activation if this is clearly isolated corruption with no signs of malicious access, no credential abuse, and no lateral movement. But be careful: many teams misclassify the early stage because they only look at the first broken app server.

Trade-offs

The big trade-off is speed versus certainty. Fast restore into the original environment feels good and is often wrong if identity, management plane, or backup credentials were touched.

In practice

Quarantine a Linux host with nftables

⚠️ This can lock you out remotely. Add your IR jump host allow rule first, and test in a non-production shell if possible.

sudo nft add table inet quarantine
sudo nft 'add chain inet quarantine input { type filter hook input priority 0; policy drop; }'
sudo nft 'add chain inet quarantine output { type filter hook output priority 0; policy drop; }'
sudo nft add rule inet quarantine input iif lo accept
sudo nft add rule inet quarantine output oif lo accept
sudo nft add rule inet quarantine input ip saddr 203.0.113.10 tcp dport 22 accept
sudo nft add rule inet quarantine output ip daddr 203.0.113.10 tcp sport 22 accept
sudo nft list ruleset

This creates a quarantine policy that drops all traffic except loopback and SSH to/from one IR source. Gotcha: if your SSH session is NATed through a different source IP than you think, the session dies immediately.

Verify a Postgres base backup on an isolated restore host

export PGDATA=/var/lib/postgresql/16/main
sudo systemctl stop postgresql
sudo rm -rf "$PGDATA"/*
sudo -u postgres tar -xzf /restore/base_2026-10-29.tar.gz -C "$PGDATA"
sudo -u postgres test -f "$PGDATA/backup_label" && echo "base backup metadata present"
sudo systemctl start postgresql
sudo -u postgres psql -c "select now(), pg_is_in_recovery();"
sudo -u postgres psql -c "select datname from pg_database;"

This restores a base backup onto a clean host and confirms the server starts. Gotcha: a base backup without required WAL segments may start failing with messages like requested WAL segment ... has already been removed; that means the backup set is incomplete for point-in-time recovery.

Freeze a systemd timer or cron-driven backup job

sudo systemctl list-timers --all | grep -i backup
sudo systemctl stop backup.timer backup.service
sudo systemctl disable backup.timer
systemctl status backup.timer --no-pager

This stops scheduled backup execution so encrypted data or deletions do not propagate further. Gotcha: disabling the timer is persistent across reboot; remember to re-enable it during controlled recovery, not ad hoc.

Further reading

• NIST SP 1800-25: Data Integrity — Identifying and Protecting Assets Against Ransomware and Other Destructive Events
• CISA and MS-ISAC: Joint Ransomware Guide
• PostgreSQL Documentation: Continuous Archiving and Point-in-Time Recovery (PITR)
• The "Disaster Recovery" and "Backups" sections of your cloud provider's architecture guidance
• Google SRE Book: Emergency Response

This article was written by an AI system and published pending human review. Verify anything you intend to act on.