Why aren't we getting prompt-caching discounts?

What does this look like in production?

• Provider usage data shows near-zero cache-read tokens relative to total input tokens, despite sending largely similar prompts repeatedly
• The system prompt and tool definitions are effectively static, yet caching discounts aren't showing up in the bill
• Something like a timestamp, request ID, or session-specific greeting appears near the start of the prompt rather than the end
• Retrieved or dynamically ordered content (RAG chunks, tool lists) is placed before the static instructions in the prompt template

Why does it happen?

Prompt caching works by recognizing that a request starts the same way as a previous request. It then skips reprocessing of that shared prefix. That only works if the prefix is actually identical, token for token. A prompt template can put something volatile near the front. Examples are a timestamp, a per-request UUID, an early user name, an unstable tool order, RAGRetrieval-Augmented Generation (RAG) retrieves relevant passages at query time and feeds them into an LLM's context. content placed before the instructions. Any of these changes the first tokens of every request. That invalidates the cache for the entire prefix, even if everything after that point is unchanged.

This is easy to miss because the prompt looks "basically the same" to a human skimming it, and the system still works correctly. It's purely a cost and latency defect, not a correctness one, which is exactly why it can go unnoticed for a long time.

How do you confirm it's this?

• Read cache-read versus cache-write (or total) token counts directly from your provider's usage data or API response metadata. A healthy cached prefix shows a high cache-read share on repeat calls
• Diff the first few hundred tokens of two consecutive requests to the same feature and check whether they are byte-identical
• Check your prompt template for anything inserted before the static system instructions: timestamps, IDs, per-user content, dynamically ordered lists
• Confirm tool definitions are serialized in a stable, canonical order rather than however they happen to be registered per call

How do you fix it?

1. Put static content first, volatile content last

   Order the prompt as: system instructions, tool definitions, few-shot examples, then anything that changes per request (retrieved chunks, user input, timestamps). The cache only needs the front to be stable.

2. Strip timestamps and UUIDs from the prefix

   If a timestamp or request ID does not need to be in the prompt, remove it. If it does, move it as late in the prompt as the model allows.

3. Canonicalize tool and few-shot ordering

   Serialize tool definitions and any few-shot examples in a fixed, deterministic order every time. Do not let registration sequence or map iteration set the order.

4. Track cache hit rate as a KPI

   Log and dashboard the cache-read share of input tokens per feature, the same way you track latency or error rate. A caching regression then triggers a specific alert before the bill creeps up.

What this doesn't cover

This applies to providers and setups that support prompt caching at all. Not every model or self-hosted setup supports it. Where it is not available, the fix is a different lever, such as shorter prompts or smaller models. Prefix ordering will not help.