How do you get the discount that prompt caching promises?

What problem does this solve?#

Prompt caching recognizes that the start of a request matches the start of an earlier request. It then skips reprocessing of that shared prefix. A prompt template can put something volatile near the front. A timestamp or a per-request ID has this effect. A user name inserted early has it too. A tool list with an unstable serialization order also breaks the prefix. So does retrieved RAGRetrieval-Augmented Generation (RAG) retrieves relevant passages at query time and feeds them into an LLM's context. content placed before the static instructions. Any of these changes the first tokens of every request. The cache then misses for everything after that point, even when the rest is unchanged.

This is easy to miss. A human skimming the prompt sees it as basically the same, and the system still works correctly. The defect affects cost and latency only, and correctness is untouched. For that reason it goes unnoticed until someone reads cache-hit-rate data directly.

How does it work?#

The team orders the prompt template deliberately. System instructions come first. Tool definitions follow in a fixed, deterministic order that never depends on map iteration or registration sequence. Few-shot examples come next. Anything that varies per call goes last. That includes user input and retrieved content. A timestamp goes there too, if the prompt needs one at all.

The team removes volatile content that the prompt does not need. It pushes the remaining volatile content as late as the model allows. No template-authoring convenience justifies placing it ahead of the static instructions. The team canonicalizes tool and few-shot ordering once. Incidental object or map ordering can change between deploys without warning, so the order must not depend on it.

The team tracks the cache-read share of input tokens per feature as a first-class KPI. It puts that share on a dashboard, as it would latency or error rate. A caching regression then raises a specific, immediate alert. An example is a new dynamic field that slips into the prefix. This beats discovering a slow, unexplained bill increase weeks later.

What are the moving parts?#

1. Order the prompt invariant-first

   Place system instructions first, then tool definitions, then few-shot examples, then per-request content.

2. Serialize tools and few-shot examples deterministically

   Fix the order for every call. Map iteration and registration sequence play no part.

3. Remove volatile tokens that do not need to be there

   Drop timestamps and request IDs from the prefix wherever the model does not need them.

4. Push necessary dynamic content as late as possible

   Place variable content only as early as the model needs it for its reasoning.

5. Track cache-read share as a dashboarded KPI

   Monitor this share per feature, as you monitor latency or error rate.

6. Alert on cache-hit-rate drops

   Treat a caching regression as a latency spike that needs a response.

When does it fit, and when doesn't it?#

What does it cost?#

How do you know it's working?#

• Cache-read token share of total input tokens, tracked continuously per feature
• Byte-identical-prefix check between consecutive same-feature requests, sampled periodically
• Measured inference costInference cost is what it costs to run a trained model on a request. compared with modeled cost, watched for the gap this pattern closes
• Alert firing rate on cache-hit-rate drops, to confirm the alert is not noise

What failure modes does this prevent?#