Find the p99-owning span in a distributed trace fast

TL;DR — To find the span that owns your p99, do not pick the longest-looking span in the trace tree. Start from a known-slow request, open one trace near the p99, then identify the span on the critical path with the largest self time or repeated contribution across many slow traces; that is usually the real owner.
Reading time: ~5 min

Goal

By the end, you will be able to take a slow request or job, open a representative distributed trace, identify the span whose latency contribution is actually driving the p99, and confirm it with a second trace or query so you can fix the right service, query, or downstream call.

Prerequisites

• Access to your tracing backend UI or API with permission to search traces and inspect spans
• A service name and operation/route name that is currently slow
• A time window where the p99 regression is visible
• curl and jq installed; check with:

curl --version
jq --version

• Optional but useful: access to logs or metrics for the same service
• If your tracer supports span attributes/events, know the key names your team uses for route, db statement, peer service, and status code

Steps

Step 1: Pin down the exact slow operation and time window

Use your metrics system or tracing search to get one concrete operation and a narrow window. If you have an HTTP service, start with route-level latency.

printf '%s
' 'service=checkout-api' 'operation=POST /orders' 'window=2026-08-10T13:00Z..2026-08-10T13:15Z' 'target=p99'

If your tracing backend has an API, fetch recent slow traces for that operation. Replace the URL and query params with your backend’s actual API shape.

curl -sG 'https://tracing.example.internal/api/traces' \
  --data-urlencode 'service=checkout-api' \
  --data-urlencode 'operation=POST /orders' \
  --data-urlencode 'start=2026-08-10T13:00:00Z' \
  --data-urlencode 'end=2026-08-10T13:15:00Z' \
  --data-urlencode 'min_duration_ms=1200' | jq '.traces[0:5][] | {trace_id, duration_ms}'

You should see a handful of trace IDs with durations clustered around your p99, not just the single worst outlier.

Step 2: Open one trace near the p99, not the absolute max

Pick a trace close to the p99 value, usually between p95 and p99.9, because the single slowest trace is often a one-off timeout, retry storm, or cold start.

curl -sG 'https://tracing.example.internal/api/traces' \
  --data-urlencode 'service=checkout-api' \
  --data-urlencode 'operation=POST /orders' \
  --data-urlencode 'start=2026-08-10T13:00:00Z' \
  --data-urlencode 'end=2026-08-10T13:15:00Z' \
  --data-urlencode 'limit=20' | jq '.traces | sort_by(.duration_ms) | .[-3] | {trace_id, duration_ms}'

Example output shape:

{
  "trace_id": "7f4f5d4a9c1e2b33",
  "duration_ms": 1387
}

You should have one trace ID whose duration is representative of the p99 you are trying to explain.

Step 3: Read the trace as a critical path, not as a flat list of long spans

Fetch the spans for that trace and sort them by start time. You are looking for the path that determines end-to-end completion, not every expensive child in parallel branches.

TRACE_ID='7f4f5d4a9c1e2b33'
curl -s "https://tracing.example.internal/api/traces/${TRACE_ID}" | jq '.spans | sort_by(.start_time_unix_nano)[] | {span_id, parent_span_id, service: .service_name, name, duration_ms, start: .start_time_unix_nano}'

Example output shape:

{
  "span_id": "a1",
  "parent_span_id": null,
  "service": "edge",
  "name": "POST /orders",
  "duration_ms": 1387,
  "start": 1754830800123456789
}
{
  "span_id": "b2",
  "parent_span_id": "a1",
  "service": "checkout-api",
  "name": "CreateOrder",
  "duration_ms": 1368,
  "start": 1754830800130000000
}
{
  "span_id": "c3",
  "parent_span_id": "b2",
  "service": "payments",
  "name": "POST /authorize",
  "duration_ms": 842,
  "start": 1754830800200000000
}

You should be able to visually trace one parent→child chain that stretches almost the full request duration.

Step 4: Calculate self time before blaming a span

A span with long duration is often just waiting on children. The owner is usually the span with large self time on the critical path, or a child span that repeatedly dominates many slow traces.

Use this formula per span:

• self_time = span.duration - sum(overlap_of_direct_children_within_span)

If your backend exposes self time directly, use that field. If not, export the trace JSON and compute it locally. This jq is a quick approximation when direct children do not overlap each other heavily:

curl -s "https://tracing.example.internal/api/traces/${TRACE_ID}" | jq '
  .spans as $s |
  $s[] |
  . as $span |
  ($s | map(select(.parent_span_id == $span.span_id) | .duration_ms) | add // 0) as $child_ms |
  {service: .service_name, name, duration_ms, child_ms: $child_ms, self_ms: (.duration_ms - $child_ms)}
'

Example output shape:

{
  "service": "checkout-api",
  "name": "CreateOrder",
  "duration_ms": 1368,
  "child_ms": 1294,
  "self_ms": 74
}
{
  "service": "payments",
  "name": "POST /authorize",
  "duration_ms": 842,
  "child_ms": 0,
  "self_ms": 842
}

You should now see which span is actually spending time itself versus merely enclosing slower children.

Step 5: Check whether the suspect span is on the critical path in multiple slow traces

One trace is not enough. Pull 5-10 traces around the p99 and look for the same service/span name repeatedly dominating self time or appearing on the longest parent→child chain.

for id in $(curl -sG 'https://tracing.example.internal/api/traces' \
  --data-urlencode 'service=checkout-api' \
  --data-urlencode 'operation=POST /orders' \
  --data-urlencode 'start=2026-08-10T13:00:00Z' \
  --data-urlencode 'end=2026-08-10T13:15:00Z' \
  --data-urlencode 'min_duration_ms=1200' | jq -r '.traces[0:5][].trace_id'); do
  echo "TRACE=$id"
  curl -s "https://tracing.example.internal/api/traces/${id}" | jq -r '.spans[] | [.service_name, .name, (.duration_ms|tostring)] | @tsv' | sort -k3 -nr | head -5
  echo
 done

You should see the same span or service show up repeatedly in the slow set. If the top span changes every trace, you probably have a broad saturation issue or bad sampling, not one owner.

Step 6: Confirm with span attributes before handing off a fix

Inspect the suspect span’s tags/attributes for the concrete thing to change: SQL text fingerprint, peer service, HTTP target, retry count, status code, queue topic, or cache key pattern.

curl -s "https://tracing.example.internal/api/traces/${TRACE_ID}" | jq '.spans[] | select(.service_name=="payments" and .name=="POST /authorize") | {service_name, name, status: .status.code, attributes}'

Example output shape:

{
  "service_name": "payments",
  "name": "POST /authorize",
  "status": "OK",
  "attributes": {
    "http.method": "POST",
    "http.route": "/authorize",
    "net.peer.name": "payments.internal",
    "retry.count": 2
  }
}

You should have one actionable owner statement such as: payments POST /authorize owns checkout-api POST /orders p99; repeated retries add ~800ms on the critical path.

Verify it works

Use both a trace-level and aggregate check.

1. Open two or three traces near the p99 and verify the same span is on the critical path with the largest self-time contribution.
2. Query the suspect span or downstream operation directly in your tracing backend for the same time window.

Example API check:

curl -sG 'https://tracing.example.internal/api/spans' \
  --data-urlencode 'service=payments' \
  --data-urlencode 'operation=POST /authorize' \
  --data-urlencode 'start=2026-08-10T13:00:00Z' \
  --data-urlencode 'end=2026-08-10T13:15:00Z' | jq '{p50_ms, p95_ms, p99_ms, error_rate}'

Expected output shape:

{
  "p50_ms": 110,
  "p95_ms": 420,
  "p99_ms": 860,
  "error_rate": 0.02
}

This proves the suspected span itself is slow at the same percentile window, not just present inside a slow parent.

Common pitfalls

Picking the longest span in the tree

Mistake: You blame the span with the biggest duration number.

Symptom: The span is a parent wrapper with tiny self time, and fixing it changes nothing.

Fix: Calculate or display self time and inspect only spans on the critical path.

Using the single worst trace as your reference

Mistake: You analyze the max-duration trace.

Symptom: You end up chasing a timeout, deploy cold start, GC pause, or one-off retry storm that does not explain the p99.

Fix: Pick a trace near p99 and compare 5-10 similar slow traces.

Ignoring parallel branches

Mistake: You add up all child durations under a parent and assume they all contributed to wall-clock latency.

Symptom: Child durations exceed parent duration or point to the wrong branch.

Fix: Follow the branch that finishes last; only that branch is on the critical path.

Trusting incomplete traces from head-based sampling

Mistake: You assume missing spans mean no downstream issue.

Symptom: Parent spans show large unexplained self time, but logs show downstream calls happened.

Fix: Check your sampling mode; if possible, inspect unsampled logs/metrics or use tail-based sampling for slow traces.

Blaming errors when the p99 is actually retries

Mistake: You filter only on failed spans.

Symptom: Most slow traces are status OK but contain retry attributes or repeated downstream calls.

Fix: Query attributes like retry.count, attempt number, or duplicate child spans under the same parent.

Mixing route-level and operation-level names

Mistake: You search for POST /orders in one service and CreateOrder in another as if they are the same span.

Symptom: Your trace set is inconsistent and comparisons are noisy.

Fix: Lock the search to one service plus one operation naming convention, then follow child spans from there.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.