Cut LLM inference cost with prompt caching, restructuring, and routing

TL;DR — If your app sends the same long instructions, policy text, tool schemas, or document context on every request, your biggest win is usually to split stable prompt prefixes from volatile user input, hash and cache the stable part, and only send the delta. Then route simple tasks to a cheaper model and reserve the expensive one for genuinely hard cases. Reading time: ~7 min

What it is and where it sits

This is not one trick; it is a small architecture pattern around your LLM call path:

1. Prompt caching: avoid paying repeatedly for the same prompt prefix or retrieved context.
2. Prompt restructuring: rewrite prompt assembly so the reusable parts are actually identical byte-for-byte and cacheable.
3. Routing by task complexity: send cheap, short, low-risk tasks to a smaller/faster model and escalate only when needed.

In a typical app, this sits between your API/controller layer and the model provider SDK. It often replaces the naive pattern of “concatenate everything into one giant prompt and always call the best model.”

Typical request flow:

Client -> App API -> Prompt Builder -> Cache Lookup -> Complexity Router -> Model Call
                    |                |                |
                    |                |                +-> small model
                    |                +-> hit: reuse prefix/context
                    +-> fetch docs/tools/policies       +-> large model

A more realistic flow with retrieval:

User request
   |
   v
HTTP handler
   |
   +--> retrieve tenant policy / docs / tool schemas
   |
   v
prompt assembler
   |
   +--> canonicalize stable prefix
   |
   +--> sha256(prefix) -> cache key
   |
   +--> cache hit? yes -> attach cached prefix handle or reused text
   |                 no -> store canonical prefix
   |
   +--> estimate complexity
   |
   +--> route to small or large model
   |
   v
response + metrics (tokens, latency, route, cache_hit)

What talks to it:

• Your web/API layer.
• Retrieval code pulling docs from Postgres, object storage, or a vector store.
• Optional policy/tool registries.
• Metrics/logging.

What it replaces:

• Rebuilding the same 5-50 KB system prompt every call.
• Sending every request to the top-tier model.
• Letting prompt formatting drift so semantically identical prompts miss cache.

Where it lives:

• Usually inside your application service, not as a separate network hop at first.
• At higher volume, teams often factor it into a dedicated “LLM gateway” service so multiple apps share cache keys, routing rules, and metrics.

How it actually works

The mechanism is simple: make the reusable part of the prompt deterministic, identify it with a stable key, and route after a cheap complexity check.

End-to-end example: support ticket triage with knowledge-grounded replies

Assume you run a SaaS support assistant. Every request currently sends:

• a 2,500-token system prompt with style, escalation policy, and compliance text,
• 8 tool schemas,
• 3 retrieved KB chunks,
• the user message.

Naive behavior: every ticket goes to the expensive model with the full assembled prompt.

Improved flow:

1. Separate stable and volatile segments

   • Stable: support policy, tone rules, tool schemas, output JSON schema.
   • Semi-stable: KB chunks for a given product/version/article set.
   • Volatile: user message, account state, current ticket thread.

2. Canonicalize the stable prefix

   • Sort JSON object keys before serialization.
   • Normalize whitespace and line endings.
   • Pin section order.
   • Strip timestamps, request IDs, and anything else that changes every call.

   If you leave generated_at: 2026-08-10T... in the system prompt, your cache hit rate goes to zero.

3. Generate cache keys

   • prefix_key = sha256(canonical_system_prefix)
   • kb_key = sha256(product_id + version + sorted(chunk_ids))

4. Cache lookup

   • If your provider supports prompt/prefix caching, send the exact same prefix bytes so the provider-side cache can hit.
   • If not, cache locally so your app reuses assembled prompt segments and avoids repeated retrieval/serialization work. This does not remove model token cost by itself, but it still cuts CPU, DB, and latency.

5. Complexity routing Run a cheap classifier before the main model call. This can be rules-first:

   • If the user asks for sentiment, tagging, language detection, summarization under 300 words, or FAQ lookup: route to small model.
   • If the request needs multi-step reasoning, policy interpretation, tool use, or contains legal/billing escalation terms: route to large model.
   • If confidence from the small model is below threshold, retry on large.

6. Call the model

   • Small model gets the same stable prefix but a tighter token budget and stricter output schema.
   • Large model gets full context only when needed.

7. Record metrics Log at minimum:

   • cache_hit_prefix
   • cache_hit_kb
   • route=small|large
   • input/output tokens
   • latency
   • fallback count
   • task type

A useful log shape:

{"req_id":"r_8f2","task":"ticket_triage","route":"small","cache_hit_prefix":true,"cache_hit_kb":true,"input_tokens":842,"output_tokens":96,"latency_ms":640,"fallback":false}

And a fallback case:

{"req_id":"r_8f3","task":"refund_policy_exception","route":"small","cache_hit_prefix":true,"cache_hit_kb":true,"input_tokens":901,"output_tokens":41,"latency_ms":410,"fallback":true,"fallback_reason":"low_confidence_schema_field=confidence score=0.42"}

The key point: prompt caching only works if the reusable prefix is truly stable, and routing only saves money if you measure misroutes and fallback rates.

When to use it (and when not to)

Use this when repeated prompt structure dominates your token bill or when your task mix has obvious easy vs hard requests.

You probably don’t need this if:

• your prompts are already short,
• your monthly token spend is trivial compared to engineering time,
• your workload is uniformly complex,
• or you have no metrics pipeline to validate quality after routing.

Trade-offs

Every benefit here has a cost.

Important edge cases:

• Multi-tenant leakage: never share cached retrieved context across tenants unless the cache key includes tenant identity and authorization scope.
• Stale policy text: if legal/compliance instructions change, old cache entries can persist. Version your prefix explicitly, for example support_policy:v17.
• Tool schema drift: changing a tool description by one character can invalidate the entire prefix cache. That is good for correctness, bad for hit rate.
• Router oscillation: if you route based on vague heuristics, you can bounce too much to fallback and lose the savings.

In practice

Example 1: Canonicalize prompt segments and cache by hash in Node.js

import crypto from "node:crypto";
import Redis from "ioredis";

const redis = new Redis(process.env.REDIS_URL);

function stableJson(value) {
  if (Array.isArray(value)) return `[${value.map(stableJson).join(",")}]`;
  if (value && typeof value === "object") {
    return `{${Object.keys(value).sort().map(k => `${JSON.stringify(k)}:${stableJson(value[k])}`).join(",")}}`;
  }
  return JSON.stringify(value);
}

function canonicalizeText(s) {
  return s.replace(/\r\n/g, "\n").replace(/[ \t]+\n/g, "\n").trim();
}

function buildStablePrefix({policyText, tools, outputSchema, version}) {
  return [
    `prefix_version=${version}`,
    "[POLICY]",
    canonicalizeText(policyText),
    "[TOOLS]",
    stableJson(tools),
    "[OUTPUT_SCHEMA]",
    stableJson(outputSchema)
  ].join("\n");
}

function sha256(s) {
  return crypto.createHash("sha256").update(s).digest("hex");
}

export async function getPrefixHandle(input) {
  const prefix = buildStablePrefix(input);
  const key = `llm:prefix:${sha256(prefix)}`;
  const cached = await redis.get(key);
  if (cached) return { key, prefix, cacheHit: true };
  await redis.set(key, prefix, "EX", 86400);
  return { key, prefix, cacheHit: false };
}

This builds a deterministic prefix and stores it in Redis keyed by SHA-256. The gotcha: if tools contains arrays whose order is not stable across deploys, your hit rate drops even though the tools are “the same” semantically.

Example 2: Route by cheap heuristics first, then fallback on low confidence

import re

SIMPLE_PATTERNS = [
    re.compile(r"\b(classify|tag|extract|translate|summarize)\b", re.I),
    re.compile(r"\b(sentiment|language|priority|category)\b", re.I),
]
COMPLEX_PATTERNS = [
    re.compile(r"\b(refund exception|legal|contract|security incident|root cause)\b", re.I),
    re.compile(r"\b(compare|reason|why|trade-off|step-by-step)\b", re.I),
]


def route_task(user_text: str, has_tools: bool, retrieved_chunks: int) -> str:
    if any(p.search(user_text) for p in COMPLEX_PATTERNS):
        return "large"
    if has_tools or retrieved_chunks > 4:
        return "large"
    if len(user_text) < 300 and any(p.search(user_text) for p in SIMPLE_PATTERNS):
        return "small"
    return "large"


def should_fallback(model_json: dict) -> bool:
    if model_json.get("confidence", 1.0) < 0.75:
        return True
    if model_json.get("needs_human_escalation") is True:
        return True
    return False

This is intentionally boring and auditable. The gotcha: do not start with a learned router unless you already have labeled traffic and offline evals; a bad router is harder to debug than a simple rule set.

Example 3: Inspect cache effectiveness from logs with jq

jq -r 'select(.task=="ticket_triage") | [.route, .cache_hit_prefix, .input_tokens, .latency_ms, .fallback] | @tsv' app.log | column -t

Typical output shape:

small  true   842   640  false
small  true   901   410  true
large  false  3120  2280 false
small  true   790   590  false

This lets you spot the obvious failure mode: lots of cache_hit_prefix=false after a deploy usually means prompt formatting changed. The gotcha: if your logs include raw prompt text, scrub PII before shipping them to centralized logging.

⚠️ If you change prompt assembly in production, you can silently change model behavior and break downstream parsers. Roll out behind a feature flag, keep the old and new prompt builders side by side, and compare schema-valid rate, fallback rate, and task success before deleting the old path.

A practical rollout sequence:

1. Add token/latency logging around current calls.
2. Split prompt into stable prefix and volatile suffix.
3. Canonicalize and hash the prefix.
4. Deploy with metrics only; verify hit rate.
5. Add a conservative router for one low-risk task class.
6. Enable fallback to large model on low confidence or parse failure.
7. Review weekly: savings, fallback rate, and quality incidents.

Further reading

• OpenAI Cookbook — prompt caching patterns
• Anthropic documentation — prompt caching and prompt design
• MDN Web Docs — HTTP Caching
• Designing Data-Intensive Applications
• Google SRE Book — Service Level Objectives

This article was written by an AI system and published pending human review. Verify anything you intend to act on.