Personal data in production logs: remove it and stop it recurring
For developers handling an incident where emails, tokens, phone numbers, or request bodies have landed in production logs. This runbook shows how to confirm the leak path, scrub existing logs with minimal blast radius, and stop the data at the source in app, proxy, and observability config.
TL;DR — You found personal data in production logs. First stop new leakage by disabling verbose/request-body logging and adding field redaction in the app/logger or collector; then rotate/expire affected logs and purge indexed copies in your log backend. The most common root cause is debug-level request logging or structured logs that serialize full request/response objects. Reading time: ~6 min
The scenario
It is 3:40 PM on a Tuesday. You are checking an error in production and a log search for a user email returns dozens of hits: full signup payloads, password reset tokens in query strings, maybe a phone number in a stack trace context object. The app is still serving traffic, support is asking whether this is reportable, and every minute you leave it running creates more copies across app logs, reverse proxy logs, and your log aggregator. You need to stop the leak path first, then clean up what already spread.
Symptoms
- Log lines containing direct identifiers or secrets, for example:
{"level":"info","msg":"request completed","path":"/signup","body":{"email":"alice@example.com","phone":"+15551234567"}}
- Access logs capturing query strings with tokens:
203.0.113.10 - - [30/Oct/2026:15:31:22 +0000] "GET /reset?token=eyJhbGciOi...&email=alice@example.com HTTP/1.1" 200 4187 "-" "Mozilla/5.0"
- Exception logs serializing request/context objects:
Error: validation failed
at createUser (...)
context={"request":{"headers":{"authorization":"Bearer eyJ..."},"body":{"ssn":"123-45-6789"}}}
- Log backend searches for an email/phone/token return hits in multiple indexes/streams.
- Users may report seeing their own email in support screenshots or exported logs.
- Container/stdout logs continue growing after deploy because the noisy logger is still enabled.
Likely causes
| Cause | How common | Quick check |
|---|---|---|
| App logger is serializing full request/response objects or bodies at info/debug level | Very common | ```bash\ngrep -RInE 'req(uest)?.body |
| Reverse proxy/access logs include full URI query strings or sensitive headers | Common | ```bash\ngrep -RInE 'log_format |
| Error reporting/middleware attaches request context with headers/body | Common | ```bash\ngrep -RInE 'captureException |
| Log shipper/collector is forwarding raw logs without redaction | Sometimes | ```bash\ngrep -RInE 'redact |
| Temporary debug flags were enabled in production | Sometimes | ```bash\nprintenv |
Step-by-step diagnosis
- Search for one known leaked value across all log stores.
EMAIL='alice@example.com'; grep -RIn --binary-files=without-match "$EMAIL" /var/log 2>/dev/null | head -50
If you get hits in local app/proxy logs, leakage is still on the host. If only your log backend has hits, jump to the collector/backend checks in step 5 and fix sections for cleanup.
- Check whether new requests are still leaking right now.
curl -sS -X POST https://app.example.com/signup -H 'Content-Type: application/json' -d '{"email":"redaction-test@example.com","phone":"+15550001111"}' -o /dev/null -w '%{http_code}\n'
sleep 2
grep -RIn 'redaction-test@example.com\|15550001111' /var/log /srv/app/logs 2>/dev/null
If the grep returns matches, this is an active leak. Go to the relevant fix section immediately before doing cleanup.
- Check app log level and request logging middleware.
printenv | grep -E 'LOG_LEVEL|DEBUG|TRACE|NODE_ENV|RAILS_LOG_LEVEL|DJANGO_LOG_LEVEL'
grep -RInE 'pino-http|morgan|winston|bunyan|lograge|structlog|logging\.Filter|req\.body|request\.body|response\.body' .
This is your problem if you find request/response objects logged directly, or production log level set to debug/trace. Jump to "App logger is serializing full request/response objects" or "Temporary debug flags".
- Check reverse proxy access log format.
nginx -T 2>/dev/null | grep -nE 'log_format|access_log|\$request_uri|\$args|\$http_authorization|\$http_cookie|\$request'
This is your problem if the format includes $request_uri, $args, $request, $http_authorization, or $http_cookie. $uri is usually safer than $request_uri because it excludes the query string. Jump to the proxy fix.
Typical bad output shape:
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$request_uri" "$http_authorization"';
- Check error reporting and collectors for attached request data.
grep -RInE 'captureException|beforeSend|setUser|setContext|extra:|request: true|send_default_pii|processors|redact|mask|transform' . /etc/fluent* /etc/vector /etc/filebeat /etc/otel* 2>/dev/null
This is your problem if exceptions include full request context, or collectors have no masking/redaction stage. Jump to the relevant fix.
- Confirm where copies exist before deletion.
printf '%s\n' 'Check: app logs, proxy logs, container stdout, node journal, object storage archives, log backend indexes/snapshots.'
journalctl --since '2 hours ago' | grep -n 'redaction-test@example.com' | head
If the same value appears in multiple places, plan cleanup in that order: live source first, then local retention, then backend indexes/archives.
View as text
[Client] ---- 1. POST /signup ----> [Proxy] | | | v | [Access log] | | v v [App] ---- 2. structured log ----> [Stdout/File] | v [Collector] ---- 3. forward/index ----> [Log backend]
Fixes
App logger is serializing full request/response objects or bodies
Stop logging bodies and sensitive headers; keep method, route, status, request id, and latency.
Node.js with pino-http:
const pinoHttp = require('pino-http');
app.use(pinoHttp({
level: process.env.LOG_LEVEL || 'info',
redact: {
paths: [
'req.headers.authorization',
'req.headers.cookie',
'req.body.password',
'req.body.token',
'req.body.email',
'res.headers["set-cookie"]'
],
censor: '[REDACTED]'
},
serializers: {
req(req) { return { method: req.method, url: req.route?.path || req.url, id: req.id }; },
res(res) { return { statusCode: res.statusCode }; }
}
}));
Python logging filter example:
class RedactFilter(logging.Filter):
KEYS = {"authorization", "cookie", "password", "token", "email", "phone"}
def filter(self, record):
if isinstance(record.args, dict):
record.args = {k: ("[REDACTED]" if k.lower() in self.KEYS else v) for k, v in record.args.items()}
return True
Verify it worked:
curl -sS -X POST https://app.example.com/signup -H 'Content-Type: application/json' -d '{"email":"redaction-test@example.com","password":"secret"}' -o /dev/null
grep -RIn 'redaction-test@example.com\|secret' /var/log /srv/app/logs 2>/dev/null
Expected: no matches.
Reverse proxy/access logs include full URI query strings or sensitive headers
Use a log format that excludes query strings and never logs auth/cookie headers.
nginx safer access log:
log_format main_safe '$remote_addr - $remote_user [$time_local] '
'"$request_method $uri $server_protocol" $status $body_bytes_sent '
'rt=$request_time ua="$http_user_agent" ref="$http_referer"';
access_log /var/log/nginx/access.log main_safe;
Reload config:
nginx -t && nginx -s reload
If reset/login links currently put tokens in query strings, move them to POST bodies or one-time form submissions where feasible. If you cannot change flow immediately, at least stop logging query strings now.
Verify it worked:
curl -I 'https://app.example.com/reset?token=redaction-test-token&email=redaction-test@example.com'
tail -n 5 /var/log/nginx/access.log
Expected log shape:
203.0.113.10 - - [30/Oct/2026:15:44:02 +0000] "HEAD /reset HTTP/1.1" 200 0 rt=0.021 ua="curl/8.7.1" ref="-"
Error reporting/middleware attaches request context with headers/body
Disable automatic request-body/header capture; keep request id, route, and sanitized user id only.
Generic pattern:
captureException(err, {
tags: { route: req.route?.path, request_id: req.id },
extra: undefined,
user: req.user ? { id: req.user.id } : undefined
});
If your SDK supports a pre-send hook, strip sensitive fields there:
function beforeSend(event) {
if (event.request) {
delete event.request.data;
if (event.request.headers) {
delete event.request.headers.authorization;
delete event.request.headers.cookie;
}
if (event.request.url) event.request.url = event.request.url.split('?')[0];
}
return event;
}
Verify it worked:
# Trigger a controlled error in a non-user-impacting endpoint if you have one, then inspect the emitted event payload/log
Expected: route/id present; no body, no auth header, no query string.
Log shipper/collector is forwarding raw logs without redaction
Add redaction at the collector as a backstop. This does not replace fixing the app/proxy source.
Vector example:
[transforms.redact]
type = "remap"
inputs = ["app_logs"]
source = '''
. = parse_json!(string!(.message)) ?? .
del(.req.headers.authorization)
del(.req.headers.cookie)
if exists(.req.body.email) { .req.body.email = "[REDACTED]" }
if exists(.req.body.phone) { .req.body.phone = "[REDACTED]" }
if exists(.req.body.password) { .req.body.password = "[REDACTED]" }
'''
Apply and restart the agent using your service manager.
Verify it worked:
journalctl -u vector --since '5 min ago' | tail -50
Expected: config loaded cleanly; new events arrive redacted.
Temporary debug flags were enabled in production
Set production log level back to info or warn, remove broad DEBUG=* style flags, and restart only the affected service.
printenv | grep -E 'LOG_LEVEL|DEBUG|TRACE'
# update your service env file or deployment manifest, then restart
systemctl restart myapp
systemctl show myapp --property=Environment
Verify it worked:
journalctl -u myapp -n 50 | grep -E 'DEBUG|TRACE|req\.body|response\.body'
Expected: no verbose request dumps.
Prevention
- Add CI grep checks that fail on obvious request/body logging patterns.
#!/usr/bin/env bash
set -euo pipefail
! grep -RInE 'logger\.(debug|info)\((req|request|response)|req\.body|request\.body|Authorization|Cookie' src/ .
Use as a CI step; non-zero exit blocks merge.
- Pin production log level in deployment manifests instead of inheriting defaults.
env:
- name: LOG_LEVEL
value: "info"
- name: DEBUG
value: ""
- Add canary redaction tests in staging/prod smoke checks.
MARKER="redaction-canary-$(date +%s)@example.invalid"
curl -sS -X POST https://app.example.com/test-redaction -H 'Content-Type: application/json' -d "{\"email\":\"$MARKER\"}" -o /dev/null || true
# Query your log backend for $MARKER; alert if any hit appears after 2 minutes.
- Reduce retention for raw node/container logs and disable unnecessary archives.
⚠️ Shortening retention can affect incident forensics and compliance evidence. Get sign-off if your org requires minimum retention.
journalctl --vacuum-time=7d
find /var/log/myapp -type f -name '*.log*' -mtime +7 -delete
- Prefer path-only logging at the edge and POST bodies over query-string tokens in app flows.
log_format main_safe '$remote_addr [$time_local] "$request_method $uri $server_protocol" $status';
- Add an alert on probable PII/secrets in logs. Start simple with regexes for emails, bearer tokens, and SSNs/phone numbers in your log backend query language, and page only on sustained matches to avoid noise.
This article was written by an AI system and published pending human review. Verify anything you intend to act on.
Have a project in mind?
Get an instant AI price estimate for it, or talk directly to our team.
One email a month on what we learn building with AI