How do you actually get the discount prompt caching promises?
The prompt is ordered with everything invariant across calls first — system instructions, tool definitions in a fixed serialization order, few-shot examples — and everything that changes per request — retrieved chunks, user input, timestamps — placed last. Because prompt caching only pays off when the shared prefix is byte-identical across requests, one volatile token near the front is enough to bust the cache for the entire call; ordering invariant-first is what makes the discount actually apply.
Also known as cache-friendly prompt ordering, stable-prefix ordering, prefix cache optimization
What problem does this solve?
Prompt caching works by recognizing that the start of a request is identical to the start of a previous one and skipping reprocessing of that shared prefix. A prompt template that puts anything volatile near the front — a timestamp, a per-request ID, a user's name inserted early, a tool list whose serialization order isn't stable across calls, retrieved RAG content placed before the static instructions — changes the first tokens of every request, invalidating the cache for everything after that point even if the rest is unchanged.
This is easy to miss because the prompt looks "basically the same" to a human skimming it and the system still functions correctly — it's purely a cost and latency defect, not a correctness one, which is exactly why it goes unnoticed until someone reads cache-hit-rate data directly instead of assuming it's fine.
How does it work?
The prompt template is ordered deliberately: system instructions first, then tool definitions serialized in a fixed, deterministic order that never depends on map iteration or registration sequence, then few-shot examples, and only then anything that varies per call, such as user input, retrieved content, or a timestamp if one is needed at all.
Anything volatile that doesn't need to be in the prompt gets removed outright; anything that does gets pushed as late as the model actually needs it, never placed ahead of the static instructions purely out of template-authoring convenience. Tool and few-shot ordering is canonicalized once and never left to depend on incidental object or map ordering, which can silently change between deploys.
Cache-read share of input tokens is tracked as a first-class KPI per feature, dashboarded the same way latency or error rate would be, so a regression in caching — a new dynamic field sneaking into the prefix — shows up as a specific, immediate alert rather than a slow, unexplained bill creep discovered weeks later.
What are the moving parts?
- Order the prompt invariant-first
System instructions, then tool definitions, then few-shot examples, then per-request content, in that order.
- Serialize tools and few-shot examples deterministically
Fix the order every time, never dependent on map iteration or registration sequence.
- Remove volatile tokens that do not need to be there
Drop timestamps and request IDs from the prefix outright wherever the model does not actually need them.
- Push necessary dynamic content as late as possible
Place anything genuinely variable only as early as the model's reasoning actually requires it.
- Track cache-read share as a dashboarded KPI
Monitor it per feature the same way latency or error rate is monitored.
- Alert on cache-hit-rate drops
Treat a regression in caching the same as a latency spike, not as background noise.
When does it fit, and when doesn't it?
Use it when
- The provider or platform supports prompt caching and the system prompt/tool definitions are largely static across calls
- Cache-read token share in provider usage data is unexpectedly low relative to how similar consecutive prompts actually are
- A prompt template currently places a timestamp, request ID, or per-user greeting near the start
- Call volume and prompt length are high enough that the caching discount is a material line item, not noise
Don't use it when
- The provider or self-hosted setup doesn't support prompt caching at all — reordering the prompt buys nothing; the lever there is shorter prompts or a smaller model instead
- Every call is already genuinely unique from the first token, fully personalized with no shared static prefix across any two requests — there is no stable prefix to preserve
- Prompt length is short enough that the caching discount would be negligible either way — the reordering effort isn't worth the marginal saving
What does it actually cost?
Reordering alone, with no model or infrastructure change, can unlock a caching discount that was already available but unclaimed
Existing prompt templates may need a real refactor to move dynamic content out of the prefix, which touches code across every call site
Removing timestamps and IDs from the prefix is often a pure win with no downside
Some volatile content genuinely needs to be early in the prompt for the model to reason correctly about it — not everything can move to the end without hurting output quality
Canonical tool/few-shot ordering is a one-time fix that keeps paying off
Requires discipline to maintain — a new tool added without matching the canonical serialization order silently regresses the cache hit rate again
Cache-hit-rate as a dashboarded KPI catches regressions immediately
Someone has to actually build and watch that dashboard — the discount degrading silently is exactly the failure this pattern exists to prevent, so skipping the metric defeats the point
How do you know it's working?
- Cache-read token share of total input tokens, per feature, tracked continuously
- Byte-identical-prefix check between consecutive same-feature requests, sampled periodically
- Actual vs. modeled inference cost, watched for the gap this pattern is meant to close
- Alert firing rate on cache-hit-rate drops, to confirm the alert itself isn't just noise
What failure modes does this prevent?
A Ship Audit checks which of these patterns your system actually needs, prioritized against what's most likely to break first.