Reconstruct an Incident When Required Logs Were Never Retained

TL;DR — When the application logs you need are gone, stop looking for them and pivot to surviving evidence: reverse proxy/access logs, systemd journal, database metadata, metrics, cloud audit events, backups, and client/browser artifacts. The most common root cause is simple retention misconfiguration or no central log shipping; confirm that first, then reconstruct a timeline from adjacent systems before data ages out there too. Reading time: ~6 min

The scenario

You get pulled into a Tuesday-afternoon incident review for a bug that started around 13:40. Users saw intermittent 500s and a few suspicious account changes, but by the time you open the app host, the container logs only go back 20 minutes and the rotated files are gone. The deploy happened from CI, autoscaling replaced two nodes, and nobody enabled centralized logging for this service. You still need to answer the two questions that matter: what happened, and how far did it spread.

Symptoms

• docker logs <container> shows only recent output, not the incident window.
• journalctl -u myapp --since "2026-10-30 13:00" returns no entries or only startup lines.
• /var/log/app.log.1, /var/log/app.log.2.gz do not exist, or rotation removed them.
• Object storage / log sink bucket has gaps for the affected hour.
• Reverse proxy still has requests for the window, e.g. nginx access lines like:

203.0.113.42 - - [30/Oct/2026:13:43:12 +0000] "POST /api/login HTTP/1.1" 500 182 "https://app.example.com/login" "Mozilla/5.0" rt=1.284 uct="0.003" uht="1.281"

• Database still shows writes around the incident time:

SELECT id, email, updated_at FROM users WHERE updated_at BETWEEN '2026-10-30 13:40' AND '2026-10-30 14:10' ORDER BY updated_at;

• Cloud/audit trail shows API actions but not app-level context, e.g. instance replacement, security group edits, secret access.
• Users can provide browser console/network evidence, screenshots, or request IDs, but the server-side request log line is missing.

Likely causes

Step-by-step diagnosis

1. Check whether the logs were deleted by retention/rotation.

sudo ls -lah /var/log /var/log/myapp* /var/log/*myapp* 2>/dev/null
sudo logrotate -d /etc/logrotate.conf 2>&1 | sed -n '/myapp/,+12p'

If you see aggressive settings like rotate 1, daily, maxage 1, or no matching stanza for your app log, this is your problem. Jump to Fixes → Log retention/rotation was too short or misconfigured.

1. Check whether logs were only ever local and the node/container was replaced.

docker ps -a --no-trunc
docker inspect --format '{{.Name}} {{.HostConfig.LogConfig.Type}} {{json .HostConfig.LogConfig.Config}}' <container>
kubectl get pods -n <ns> -o wide --sort-by=.status.startTime

Output like json-file {"max-file":"1","max-size":"10m"} or pods/nodes created after the incident means the old local logs likely died with the workload. Jump to Fixes → Containers/node replacement discarded local logs and continue reconstruction from remaining sources.

1. Check whether a shipper exists and whether it was actually running.

ps aux | egrep 'promtail|fluent-bit|vector|filebeat|rsyslog' | grep -v grep
systemctl --no-pager --full status promtail fluent-bit vector filebeat rsyslog 2>/dev/null

If nothing is installed/running, or service status shows repeated send failures, this is your problem. Jump to Fixes → No centralized log shipping for this service/environment.

1. Check journald/syslog persistence.

journalctl --disk-usage
grep -R '^[# ]*Storage=\|^[# ]*SystemMaxUse=\|^[# ]*RuntimeMaxUse=' /etc/systemd/journald.conf /etc/systemd/journald.conf.d 2>/dev/null
journalctl -u myapp --since '2026-10-30 13:00' --until '2026-10-30 15:00' --no-pager

If Storage=volatile is set, disk usage is tiny, or the service restarted after the incident with no older entries, jump to Fixes → Journald or syslog storage is volatile/capped.

1. Validate the incident window before you reconstruct anything.

timedatectl
date -u
sudo awk 'NR==1{print; exit}' /var/log/nginx/access.log

If app timestamps are local time but proxy/db are UTC, or NTP was unsynced, your timeline may be shifted. Jump to Fixes → Time window is wrong due to timezone/clock skew.

1. Reconstruct from adjacent systems, cheapest first.

find /var/log -maxdepth 2 -type f | egrep 'nginx|apache|haproxy|postgres|mysql|audit|auth' | sort
sudo zgrep '30/Oct/2026:13:4[0-9]\|30/Oct/2026:14:0[0-9]' /var/log/nginx/access.log*
sudo journalctl --since '2026-10-30 13:40' --until '2026-10-30 14:10' -o short-iso --no-pager | egrep 'oom|killed|segfault|denied|sudo|sshd'

If proxy, DB, or auth logs clearly cover the window, this is now an evidence-reconstruction exercise. Jump to Fixes → Evidence exists elsewhere, but only in adjacent systems.

[User Browser] ---- 1. request ----> [Reverse Proxy]
     |                                   |
     | 6. screenshots, HAR               | 2. access/error logs
     v                                   v
[Client Evidence]                   [App Service]
                                         |
                                         | 3. missing app logs
                                         v
                                   [DB / Queue / Cache]
                                         |
                                         | 4. writes, timestamps
                                         v
                                   [Audit / Metrics]

Fixes

Log retention/rotation was too short or misconfigured

Set explicit retention for the app logs and test the policy before waiting for the next incident.

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
  daily
  rotate 14
  compress
  delaycompress
  missingok
  notifempty
  copytruncate
  su myapp myapp
}

Test it:

sudo logrotate -d /etc/logrotate.d/myapp
sudo logrotate -f /etc/logrotate.d/myapp
ls -lah /var/log/myapp

Trade-off: copytruncate avoids app restart but can lose a few lines during rotation; if your app can reopen files on SIGUSR1/SIGHUP, use that instead.

Verify it worked:

sudo logrotate -d /etc/logrotate.d/myapp | grep rotating

No centralized log shipping for this service/environment

Install one shipper and send logs off-host. Generic example with rsyslog forwarding:

# /etc/rsyslog.d/60-forward.conf
*.* action(type="omfwd" target="logs.example.net" port="514" protocol="tcp" action.resumeRetryCount="-1" queue.type="linkedList")

sudo systemctl restart rsyslog
sudo systemctl status rsyslog --no-pager
logger -t retention-test "hello from $(hostname)"

If your app logs to files, also ingest those files with your chosen shipper; exact config varies by tool. The important part is: logs leave the node before rotation or termination.

Verify it worked:

logger -t retention-test "shipper-check $(date -Is)"

Then confirm the line appears in your log destination.

Containers/node replacement discarded local logs

Increase container log limits only as a short-term buffer; the real fix is off-node shipping.

docker inspect --format '{{json .HostConfig.LogConfig}}' <container>

If using Docker json-file, set saner defaults:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "5"
  }
}

sudo tee /etc/docker/daemon.json >/dev/null <<'EOF'
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "5"
  }
}
EOF
sudo systemctl restart docker

⚠️ Restarting Docker will interrupt containers on that host. For Kubernetes, deploy a node-level log collector DaemonSet appropriate for your platform and verify it tails /var/log/containers/*.log.

Verify it worked:

docker info | sed -n '/Logging Driver/p'

Journald or syslog storage is volatile/capped

Persist journald and raise caps to something that survives a normal incident review window.

# /etc/systemd/journald.conf.d/persistent.conf
[Journal]
Storage=persistent
SystemMaxUse=2G
RuntimeMaxUse=512M
MaxRetentionSec=14day

sudo mkdir -p /etc/systemd/journald.conf.d /var/log/journal
sudo systemctl restart systemd-journald
journalctl --disk-usage

Trade-off: larger journal retention consumes disk; set alerts on filesystem usage.

Verify it worked:

journalctl --list-boots | tail

Older boots should remain visible after restart.

Time window is wrong due to timezone/clock skew

Normalize all reconstruction work to UTC, then fix host time sync.

timedatectl set-timezone UTC
sudo timedatectl set-ntp true
timedatectl status

When querying logs, always use explicit offsets:

journalctl --since '2026-10-30 13:40:00 UTC' --until '2026-10-30 14:10:00 UTC' --no-pager

For SQL, do the same:

SET TIME ZONE 'UTC';
SELECT now(), current_setting('TimeZone');

Verify it worked:

timedatectl status | egrep 'Time zone|System clock synchronized'

Evidence exists elsewhere, but only in adjacent systems

Build the timeline from surviving systems before those rotate too.

1. Reverse proxy:

sudo zgrep '30/Oct/2026:13:4[0-9]\|30/Oct/2026:14:0[0-9]' /var/log/nginx/access.log* | awk '{print $4,$5,$6,$7,$9,$10}' | head -20

1. Error logs:

sudo zgrep '2026/10/30 13:|2026/10/30 14:' /var/log/nginx/error.log*

1. Database writes:

SELECT table_name FROM information_schema.columns WHERE column_name IN ('created_at','updated_at');
SELECT id, updated_at FROM users WHERE updated_at BETWEEN '2026-10-30 13:40+00' AND '2026-10-30 14:10+00' ORDER BY updated_at;

1. Auth/audit trail:

sudo journalctl --since '2026-10-30 13:40 UTC' --until '2026-10-30 14:10 UTC' -u ssh -u sudo --no-pager
ausearch -ts '2026-10-30 13:40:00' -te '2026-10-30 14:10:00' 2>/dev/null | head

1. Metrics for correlation:

curl -s http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=rate(http_requests_total[5m])' | jq '.data.result[0:3]'

1. Client evidence: ask for HAR, exact timestamp, timezone, user ID, and any X-Request-ID header.

Verify it worked:

printf 'Have: proxy=%s db=%s audit=%s metrics=%s client=%s
' yes yes yes yes yes

If you can place request spikes, writes, and infra events on one UTC timeline, you have enough for a defensible incident report even without app logs.

Prevention

• Ship logs off-host in every environment, including staging. Add a CI check that fails if no shipper config is present:

test -f /etc/rsyslog.d/60-forward.conf -o -f /etc/fluent-bit/fluent-bit.conf -o -f /etc/vector/vector.toml

• Pin retention explicitly for app logs and journald instead of relying on distro defaults:

grep -R 'rotate 14\|Storage=persistent\|MaxRetentionSec=14day' /etc/logrotate.d /etc/systemd/journald.conf.d

• Add a canary log line and alert if it does not arrive in the sink within 5 minutes:

logger -t log-canary "$(hostname) $(date -Is)"

• Preserve request correlation IDs at the edge so adjacent logs stay useful:

log_format main '$remote_addr - $request_id [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"';
proxy_set_header X-Request-ID $request_id;
add_header X-Request-ID $request_id always;

• Snapshot critical audit sources longer than app logs: cloud audit events, DB logs, auth logs. At minimum, archive rotated files nightly:

tar -czf /srv/log-archive/$(hostname)-$(date +%F).tgz /var/log/nginx /var/log/auth.log* /var/log/postgresql 2>/dev/null

• Add an incident checklist item: convert all timestamps to UTC first. Put this in the runbook template:

- Incident window in UTC:
- Evidence sources checked:
- Request IDs/user IDs involved:
- Earliest surviving log in each system:

This article was written by an AI system and published pending human review. Verify anything you intend to act on.