Why aren't we getting prompt-caching discounts?

What does this look like in production?

• Provider usage data shows near-zero cache-read tokens relative to total input tokens, despite sending largely similar prompts repeatedly
• The system prompt and tool definitions are effectively static, yet caching discounts aren't showing up in the bill
• Something like a timestamp, request ID, or session-specific greeting appears near the start of the prompt rather than the end
• Retrieved or dynamically ordered content (RAG chunks, tool lists) is placed before the static instructions in the prompt template

Why does it happen?

Prompt caching works by recognizing that the beginning of a request is identical to the beginning of a previous request, and skipping reprocessing of that shared prefix. That only works if the prefix is actually identical, token for token. A prompt template that puts anything volatile near the front — a current timestamp, a per-request UUID, a user's name inserted early, a tool list whose order isn't stable across calls, or retrieved RAG content placed before the instructions — changes the first tokens of every request, which invalidates the cache for the entire prefix even if everything after that point is unchanged.

This is easy to miss because the prompt looks "basically the same" to a human skimming it, and the system still works correctly — it's purely a cost and latency defect, not a correctness one, which is exactly why it can go unnoticed for a long time.

How do you confirm it's this?

• Read cache-read versus cache-write (or total) token counts directly from your provider's usage data or API response metadata — a healthy cached prefix shows a high cache-read share on repeat calls
• Diff the first few hundred tokens of two consecutive requests to the same feature and check whether they are byte-identical
• Check your prompt template for anything inserted before the static system instructions: timestamps, IDs, per-user content, dynamically ordered lists
• Confirm tool definitions are serialized in a stable, canonical order rather than however they happen to be registered per call

How do you fix it?

1. Put static content first, volatile content last

   Order the prompt as: system instructions, tool definitions, few-shot examples, then anything that changes per request (retrieved chunks, user input, timestamps) — the cache only needs the front to be stable.

2. Strip timestamps and UUIDs from the prefix

   If a timestamp or request ID doesn't need to be in the prompt at all, remove it; if it does, move it as late in the prompt as the model actually needs it.

3. Canonicalize tool and few-shot ordering

   Serialize tool definitions and any few-shot examples in a fixed, deterministic order every time, rather than an order that depends on registration sequence or map iteration.

4. Track cache hit rate as a KPI

   Log and dashboard the cache-read share of input tokens per feature, the same way you'd track latency or error rate, so a regression in caching shows up as a specific alert rather than a slow bill creep.

What this doesn't cover

This applies to providers and setups that support prompt caching at all — not every model or self-hosted setup does, and where it isn't available, the fix is a different lever (shorter prompts, smaller models) rather than prefix ordering.