How to read a stack trace after a container already restarted

TL;DR — When a container has already restarted, your first stop is the previous instance’s logs and state: docker logs --previous <container> or kubectl logs --previous <pod> -c <container>. The most common outcome is that the stack trace was only written to stdout/stderr and is still retrievable from the runtime; if not, check exit code 137/OOMKilled, app logging to a file inside the container, or a process that dies before flushing logs. Reading time: ~6 min

The scenario

You push a routine Tuesday deploy, CI is green, and five minutes later the API starts returning intermittent 502 from the reverse proxy. By the time you shell into the host, the container is already back up, health checks are green again, and docker logs only shows the fresh startup banner. The app definitely crashed once — maybe several times — but the stack trace you need was emitted by the previous process and is now easy to miss. You need the exact exception, exit code, and whether this was an app crash, an OOM kill, or a bad logging setup.

Symptoms

• Your app briefly disappeared, then came back on its own.
• Reverse proxy or ingress logs show upstream failures during the restart window, for example:

2026/10/01 14:07:33 [error] 412#412: *188 connect() failed (111: Connection refused) while connecting to upstream, client: 203.0.113.24, server: api.example.com, request: "GET /health HTTP/1.1", upstream: "http://172.18.0.5:3000/health", host: "api.example.com"

• Docker shows a restart count incrementing:

docker ps --format 'table {{.Names}}\t{{.Status}}'

NAMES        STATUS
api-web      Up 18 seconds (healthy)

• Kubernetes shows CrashLoopBackOff or a non-zero last exit code:

kubectl get pod api-web-7f6d9b7c9d-9xk2m

NAME                         READY   STATUS             RESTARTS   AGE
api-web-7f6d9b7c9d-9xk2m     1/1     Running            4          12m

• kubectl describe pod or docker inspect shows exit code 1, 134, 137, 139, or 143.
• You expected a stack trace, but current logs only show the new boot sequence:

Starting server on :3000
Connected to postgres
Listening for requests

• In OOM cases, you may see kernel/runtime messages instead of an app exception:

Killed

or in Kubernetes:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Likely causes

Step-by-step diagnosis

1. Check whether you need the previous instance’s logs, not the current one.

# Docker
docker ps -a --format 'table {{.Names}}\t{{.Status}}'
docker logs --timestamps --tail 200 --previous api-web

# Kubernetes
kubectl get pod api-web-7f6d9b7c9d-9xk2m
kubectl logs --timestamps --previous api-web-7f6d9b7c9d-9xk2m -c api

If you see the exception in --previous output, this is your problem. Jump to ### Previous container logs exist, but you are looking at the current instance.

1. Inspect the last exit reason and exit code.

# Docker
docker inspect api-web --format 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}} error={{.State.Error}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}'

# Kubernetes
kubectl describe pod api-web-7f6d9b7c9d-9xk2m | sed -n '/Containers:/,/Conditions:/p'

Output that means "this is your problem":

exit=137 oom=true error=

or:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Jump to ### The process was OOM-killed before a useful stack trace was emitted.

1. Verify where logs are actually going.

# Docker runtime log path and driver
docker inspect api-web --format 'logpath={{.LogPath}}'
docker info --format 'driver={{.LoggingDriver}}'

# Inside the container: common app file locations
docker exec api-web sh -lc 'ls -lah /var/log /app/logs 2>/dev/null; find /tmp -maxdepth 2 -type f \( -name "*.log" -o -name "hs_err_pid*" \) 2>/dev/null | head -50'

If stdout/stderr is sparse but app logs exist under /var/log, /app/logs, or a framework-specific path, jump to ### The app writes errors to a file inside the container instead of stdout/stderr.

1. Check whether the process died before buffers flushed or before the logger initialized.

docker inspect api-web --format 'exit={{.State.ExitCode}} error={{.State.Error}}'
kubectl logs --previous api-web-7f6d9b7c9d-9xk2m -c api --tail=20

Signals to look for:

• exit=139 → segfault
• exit=134 → abort/SIGABRT
• Empty previous logs except maybe Killed or nothing at all Jump to ### The process crashes so early or so hard that buffered logs never flush.

1. Check whether the runtime already rotated away the evidence.

# Docker JSON-file logs on the host
LOG=$(docker inspect api-web --format '{{.LogPath}}'); echo "$LOG"; sudo ls -lh "$LOG" "$LOG".* 2>/dev/null

# If using systemd/journald for container logs
sudo journalctl --since '30 minutes ago' CONTAINER_NAME=api-web

If the previous logs are missing and only tiny fresh files remain, jump to ### Old logs were rotated or discarded by the runtime/journal before you checked.

1. If you suspect a native crash, look for a core dump or language crash artifact.

sudo coredumpctl list | tail -20
docker exec api-web sh -lc 'find / -maxdepth 3 -type f \( -name "core" -o -name "core.*" -o -name "hs_err_pid*" \) 2>/dev/null | head -50'

If you find a matching artifact timestamped around the crash, jump to ### The stack trace is only available via a core dump or language-specific crash artifact.

Fixes

Previous container logs exist, but you are looking at the current instance

Use the runtime’s previous-instance log retrieval and capture it before another restart overwrites your context.

# Docker
docker logs --timestamps --previous api-web > /tmp/api-web.previous.log

# Kubernetes
kubectl logs --timestamps --previous api-web-7f6d9b7c9d-9xk2m -c api > /tmp/api.previous.log

If the pod has multiple containers, always specify -c <container>; otherwise you may read the sidecar instead of the app.

Verify it worked:

grep -E 'Exception|Traceback|panic:|Caused by:|SIGSEGV' /tmp/api.previous.log | head

The process was OOM-killed before a useful stack trace was emitted

Increase the memory limit, reduce startup memory spikes, or both. In Kubernetes:

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"

Apply it:

kubectl apply -f deployment.yaml
kubectl rollout status deploy/api-web

In plain Docker:

docker update --memory 1g --memory-swap 1g api-web

For JVM apps, cap heap below the container limit:

JAVA_TOOL_OPTIONS='-XX:MaxRAMPercentage=70 -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp'

For Node.js:

NODE_OPTIONS='--max-old-space-size=768'

Verify it worked:

kubectl describe pod <new-pod> | grep -A3 'Last State'

You want no new OOMKilled events after load returns.

The app writes errors to a file inside the container instead of stdout/stderr

Change the app/process manager to log to stdout/stderr so the container runtime preserves it across restarts. Examples:

access_log /dev/stdout;
error_log /dev/stderr warn;

# supervisord
stdout_logfile=/dev/fd/1
stdout_logfile_maxbytes=0
stderr_logfile=/dev/fd/2
stderr_logfile_maxbytes=0

# shell entrypoint: force unbuffered Python output
export PYTHONUNBUFFERED=1
exec python -u app.py

If you must keep file logs, mount persistent storage and ship them off-box.

Verify it worked:

docker logs api-web --tail 50

You should now see application errors without docker exec into the container.

The process crashes so early or so hard that buffered logs never flush

Disable buffering and emit crash output directly to stderr. Common fixes:

# Python
export PYTHONUNBUFFERED=1
exec python -u app.py

# Node.js: enable fatal reports
export NODE_OPTIONS='--report-uncaught-exception --report-on-fatalerror --report-directory=/tmp'
exec node server.js

# Go: usually already stderr-friendly; ensure your logger writes to stderr

# Java: send stack traces to stderr and generate hs_err files on fatal VM errors
export JAVA_TOOL_OPTIONS='-XX:ErrorFile=/tmp/hs_err_pid%p.log -XX:+CrashOnOutOfMemoryError'
exec java -jar app.jar

If PID 1 is a shell wrapper, replace it with exec ... so signals and exit codes propagate cleanly.

# bad
./migrate.sh
node server.js

# good
./migrate.sh
exec node server.js

Verify it worked:

docker restart api-web && sleep 2 && docker logs api-web --tail 100

You should see startup and crash output immediately, not only after process exit.

Old logs were rotated or discarded by the runtime/journal before you checked

Increase retention for container logs and centralize them. For Docker’s json-file driver:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "5"
  }
}

Write that to /etc/docker/daemon.json, then restart Docker.

⚠️ Restarting the Docker daemon can interrupt running containers depending on your host setup and restart policies. Do this in a maintenance window or on one node at a time.

sudo systemctl restart docker

If your host uses journald, raise retention in /etc/systemd/journald.conf and reload:

SystemMaxUse=2G
RuntimeMaxUse=512M
MaxRetentionSec=7day

sudo systemctl restart systemd-journald

Verify it worked:

docker info --format '{{.LoggingDriver}}'; sudo ls -lh $(docker inspect api-web --format '{{.LogPath}}')*

The stack trace is only available via a core dump or language-specific crash artifact

Collect and inspect the artifact. On systemd hosts:

sudo coredumpctl list | tail -20
sudo coredumpctl info <PID-or-EXE>
sudo coredumpctl gdb <PID-or-EXE>

For JVM fatal errors:

docker cp api-web:/tmp/hs_err_pid123.log ./hs_err_pid123.log
sed -n '1,120p' ./hs_err_pid123.log

For Node diagnostic reports:

docker exec api-web sh -lc 'ls -lah /tmp/report.*.json'
docker cp api-web:/tmp/report.2026*.json ./

If you need core dumps from containers, set a writable core pattern/location on the host and allow dump generation in your runtime/security profile; exact steps vary by distro and orchestrator.

Verify it worked:

ls -1 ./hs_err_pid*.log ./report.*.json 2>/dev/null

Prevention

• Add a one-command previous-log check to your incident docs and shell history.

# Docker alias
alias dprev='docker logs --timestamps --previous'
# Kubernetes alias
alias kprev='kubectl logs --timestamps --previous'

• Pin container log retention on every host instead of relying on defaults.

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "50m", "max-file": "5" }
}

• Fail CI if the image writes app logs to files instead of stdout/stderr. Example smoke test:

docker run --rm your-image sh -lc 'test -e /dev/stdout && ! grep -R "/var/log/app" /app /etc 2>/dev/null'

• Emit crash artifacts for runtimes that support them.

# Java
JAVA_TOOL_OPTIONS='-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:ErrorFile=/tmp/hs_err_pid%p.log'
# Node
NODE_OPTIONS='--report-uncaught-exception --report-on-fatalerror --report-directory=/tmp'

• Alert on restart count and OOM kills, not just uptime. Kubernetes example:

kubectl get events --all-namespaces --field-selector reason=OOMKilled
kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount'

Wire equivalent metrics from your runtime into Prometheus or your existing alerting stack.

• Keep entrypoints simple and use exec for the main process so signals, exit codes, and stderr are preserved.

#!/usr/bin/env sh
set -eu
./migrate.sh
exec /app/server

This article was written by an AI system and published pending human review. Verify anything you intend to act on.