Contain a compromised Kubernetes pod without destroying evidence

TL;DR — When a Kubernetes workload looks compromised, do not start with kubectl delete pod. First stop new spread: cordon the node if needed, isolate the pod with a deny-all NetworkPolicy or node-level block, and stop controllers from recreating it by scaling the owner to zero. Then collect logs, manifests, events, process lists, and a filesystem copy before you kill anything. Reading time: ~5 min

Goal

When you finish, the suspected workload is isolated from network communication and from being recreated automatically, and you have a local evidence bundle containing its manifests, logs, events, node placement, process list, and a filesystem copy you can hand to security or inspect later.

Prerequisites

• kubectl access to the cluster with permission to get, describe, logs, exec, cp, patch, scale, and apply in the target namespace — check with kubectl auth can-i get pods -n <ns>
• Kubernetes cluster version 1.24+ — check with kubectl version --short
• A shell with bash, tar, and gzip
• The target namespace and pod name — get them with kubectl get pods -A -o wide
• Enough local disk to store logs and a pod filesystem copy
• If your CNI enforces NetworkPolicy, permission to create one in the namespace; if it does not, node access for emergency host firewalling

Steps

Step 1: Record the pod, owner, and node before touching it

export NS=default
export POD=suspicious-pod-7d9c6f5b8d-x2k9m
mkdir -p evidence/$NS-$POD
kubectl get pod -n "$NS" "$POD" -o wide | tee evidence/$NS-$POD/pod-wide.txt
kubectl get pod -n "$NS" "$POD" -o yaml > evidence/$NS-$POD/pod.yaml
kubectl describe pod -n "$NS" "$POD" > evidence/$NS-$POD/pod.describe.txt
kubectl get events -n "$NS" --field-selector involvedObject.name="$POD" --sort-by=.lastTimestamp > evidence/$NS-$POD/events.txt
kubectl get pod -n "$NS" "$POD" -o jsonpath='{.metadata.ownerReferences[0].kind}{"/"}{.metadata.ownerReferences[0].name}{"\n"}{.spec.nodeName}{"\n"}' | tee evidence/$NS-$POD/owner-and-node.txt

Success looks like owner-and-node.txt containing something like ReplicaSet/web-6d4f7d9c6f on the first line and a node name on the second.

Step 2: Stop the controller from recreating the workload

⚠️ This can cause downtime for that application. Do this only for the compromised workload's owner, not the whole namespace.

OWNER_KIND=$(cut -d/ -f1 evidence/$NS-$POD/owner-and-node.txt)
OWNER_NAME=$(cut -d/ -f2 evidence/$NS-$POD/owner-and-node.txt)
if [ "$OWNER_KIND" = "ReplicaSet" ]; then
  DEPLOY=$(kubectl get rs -n "$NS" "$OWNER_NAME" -o jsonpath='{.metadata.ownerReferences[0].name}')
  kubectl scale deployment -n "$NS" "$DEPLOY" --replicas=0
  echo "deployment/$DEPLOY" | tee evidence/$NS-$POD/scaled-owner.txt
else
  kubectl scale "$OWNER_KIND" -n "$NS" "$OWNER_NAME" --replicas=0
  echo "$OWNER_KIND/$OWNER_NAME" | tee evidence/$NS-$POD/scaled-owner.txt
fi
kubectl get pods -n "$NS" -o wide

Success looks like the replica count for the owner going to 0, and no new replacement pod appearing after ~10-30 seconds.

Step 3: Isolate the pod from the network with a deny-all policy

⚠️ This only works if your CNI enforces NetworkPolicy. If it does not, go to Step 4 immediately.

kubectl get pod -n "$NS" "$POD" --show-labels

Pick one label unique to the pod template, then apply this policy with the literal label key/value from the pod output:

cat > evidence/$NS-$POD/isolate-netpol.yaml <<'YAML'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: isolate-suspicious-pod
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Ingress
  - Egress
  ingress: []
  egress: []
YAML
sed -i.bak "s/namespace: default/namespace: $NS/" evidence/$NS-$POD/isolate-netpol.yaml
kubectl apply -f evidence/$NS-$POD/isolate-netpol.yaml
kubectl describe networkpolicy -n "$NS" isolate-suspicious-pod

Success looks like networkpolicy.networking.k8s.io/isolate-suspicious-pod created and PodSelector: app=web in the describe output.

Step 4: If NetworkPolicy is not enforced, cordon the node and block egress at the host

⚠️ Host firewall rules can affect other pods on the same node. Prefer a rule targeted to the pod IP. This step requires node access and may still interrupt unrelated traffic if done sloppily. Get the node and pod IP:

kubectl get pod -n "$NS" "$POD" -o jsonpath='{.status.podIP}{"\n"}{.spec.nodeName}{"\n"}' | tee evidence/$NS-$POD/podip-and-node.txt

Prevent new scheduling to that node:

NODE=$(tail -n1 evidence/$NS-$POD/podip-and-node.txt)
kubectl cordon "$NODE"

On the node, block the pod IP with iptables:

POD_IP=$(head -n1 evidence/$NS-$POD/podip-and-node.txt)
sudo iptables -I FORWARD -s "$POD_IP" -j DROP
sudo iptables -I FORWARD -d "$POD_IP" -j DROP
sudo iptables -L FORWARD -n --line-numbers | head

Success looks like node/<name> cordoned from kubectl and two new DROP rules matching the pod IP in iptables -L output.

Step 5: Collect logs, process list, mounts, and network state

kubectl logs -n "$NS" "$POD" --all-containers=true --timestamps > evidence/$NS-$POD/logs-current.txt 2>&1 || true
kubectl logs -n "$NS" "$POD" --all-containers=true --timestamps --previous > evidence/$NS-$POD/logs-previous.txt 2>&1 || true
kubectl exec -n "$NS" "$POD" -- sh -c 'id; uname -a; ps auxww; cat /proc/1/cgroup; mount; ss -plantu || netstat -plantu' > evidence/$NS-$POD/runtime.txt 2>&1 || true

Success looks like runtime.txt containing process lines such as PID USER ... COMMAND and socket lines such as ESTAB 0 0 10.42.1.15:443 ....

Step 6: Copy the container filesystem without deleting the pod

⚠️ kubectl cp uses tar in the container. If the image does not have tar, this will fail with exec: "tar": executable file not found in $PATH.

kubectl cp -n "$NS" "$POD":/ evidence/$NS-$POD/filesystem-copy >/tmp/kubectl-cp-$POD.log 2>&1 || cat /tmp/kubectl-cp-$POD.log

If kubectl cp fails because tar is missing, stream a tarball if the container has sh and tar:

kubectl exec -n "$NS" "$POD" -- sh -c 'tar cf - / 2>/dev/null' | gzip > evidence/$NS-$POD/rootfs.tar.gz

Success looks like a populated filesystem-copy/ directory or a non-zero-size rootfs.tar.gz file.

Step 7: Freeze the final state and optionally delete only after evidence is captured

tar czf evidence/$NS-$POD-bundle.tgz -C evidence "$NS-$POD"
ls -lh evidence/$NS-$POD-bundle.tgz

If you need the pod gone after capture:

kubectl delete pod -n "$NS" "$POD" --wait=false

Success looks like a tarball on disk and, if you deleted the pod, pod "<name>" deleted with no replacement pod because the owner is scaled to zero.

Verify it works

Run these checks:

kubectl get pod -n "$NS" "$POD" -o wide
kubectl get networkpolicy -n "$NS"
kubectl get $(cat evidence/$NS-$POD/scaled-owner.txt | tr '[:upper:]' '[:lower:]') -n "$NS" -o yaml | grep -E '^  replicas: 0|^replicas: 0'
ls -lh evidence/$NS-$POD-bundle.tgz

Expected results:

NAME                               READY   STATUS    RESTARTS   AGE   IP           NODE
suspicious-pod-7d9c6f5b8d-x2k9m    1/1     Running   0          22m   10.42.1.15   worker-03

NAME                    POD-SELECTOR   AGE
isolate-suspicious-pod  app=web        12s

-rw-r--r-- 1 you you 148M Oct 30 14:22 evidence/default-suspicious-pod-7d9c6f5b8d-x2k9m-bundle.tgz

If you used host firewalling instead of NetworkPolicy, verify the node is cordoned and the rules exist:

kubectl get node "$NODE"
sudo iptables -L FORWARD -n --line-numbers | grep "$POD_IP"

You should see SchedulingDisabled on the node and DROP rules for the pod IP.

Common pitfalls

Scaling the pod instead of its owner

Mistake: running kubectl delete pod or trying kubectl scale pod .... Symptom: the pod disappears and a new one appears immediately with a different suffix. Fix: scale the owning Deployment, StatefulSet, or ReplicaSet to zero using the command in Step 2.

Applying a deny-all NetworkPolicy in a cluster that does not enforce it

Mistake: creating the policy and assuming traffic is blocked. Symptom: the policy exists, but the pod still reaches external IPs and keeps receiving traffic. Fix: use Step 4: kubectl cordon <node> and add host firewall DROP rules for the pod IP.

Using labels in the NetworkPolicy that match other pods

Mistake: selecting app=web when several healthy pods share that label. Symptom: unrelated pods lose ingress/egress at the same time. Fix: first inspect labels with kubectl get pod -n "$NS" "$POD" --show-labels and use the most specific template label available.

kubectl cp fails because the container image is minimal

Mistake: assuming every container has tar. Symptom: output like exec: "tar": executable file not found in $PATH. Fix: use kubectl exec ... -- sh -c 'tar cf - / 2>/dev/null' | gzip > rootfs.tar.gz if sh and tar exist, or collect evidence from the node/container runtime instead.

Forgetting --previous logs after a crash loop

Mistake: collecting only current logs. Symptom: logs-current.txt is short or irrelevant while the malicious activity was in the previous container instance. Fix: always run kubectl logs --previous --all-containers=true --timestamps and save both current and previous logs.

Cordoning the node but leaving the compromised pod connected

Mistake: stopping new scheduling and thinking the incident is contained. Symptom: no new pods land on the node, but the suspicious pod still has active sockets in ss -plantu. Fix: add the pod-IP-specific iptables DROP rules from Step 4 or apply an enforced deny-all NetworkPolicy.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.