Why is our inference bill three times the estimate?

What does this look like in production?

• Actual monthly inference spend runs meaningfully above the number modeled from expected traffic times per-token price
• The gap between sticker price and actual spend grows specifically on features that retry, fall back, or chain multiple calls
• Nobody can currently answer "what is our cost per successful outcome" without manually reconstructing it from raw billing data
• A recent quality-improving change (bigger model, more context, self-correction pass) coincided with a step change in spend that wasn't separately budgeted

Why does it happen?

A per-token cost estimate models one clean attempt at a task. Production doesn't run one clean attempt per outcome — it runs however many attempts it takes, including retries on rate limits or malformed responses, fallback calls to a second, often pricier model when the first provider or model fails, and, for agentic workflows, several chained calls to reach one completed task. Each of those is a real, billed call, and none of them is visible in an estimate built from "tokens per request times price per token" without also accounting for how many requests one successful outcome actually takes.

This compounds specifically for multi-step and agentic workloads, where the calls-to-completions ratio can be several to one even in a healthy system, and much worse when reliability problems elsewhere (like an unbounded agent loop) are also in play.

How do you confirm it's this?

• Measure cost per successful outcome, not cost per call — divide total spend for a feature by the count of tasks that actually completed successfully, not the count of calls made
• Track the calls-to-completions ratio over time and watch for drift, the same way you would watch an error rate
• Break down spend by call type (initial attempt, retry, fallback) to see which is actually driving the gap
• Compare the modeled per-token estimate against actual billing for the same period, tagged by feature

How do you fix it?

1. Set a budget per outcome, not per call

   Define an acceptable cost ceiling for one completed task, inclusive of retries and fallbacks, and alert when a feature approaches or exceeds it — not just when total spend crosses a threshold.

2. Cap retries explicitly

   An unbounded retry policy can silently multiply cost on a persistently failing call; a fixed retry ceiling bounds the worst case and forces a visible failure instead of an invisible cost spiral.

3. Make fallback routing explicit and measured

   Know exactly when and how often your system falls back to a different (often pricier) model or provider, and treat that rate as a tracked metric, not an invisible safety net.

4. Alert on calls-per-outcome drift

   A rising ratio of calls to completed outcomes is an early warning of a reliability regression that is also a cost regression — catch it there rather than only in the monthly invoice.

What this doesn't cover

This explains cost overruns from repeated or chained calls specifically — if spend is high because a single call's context or output size grew (not because of retries or chaining), the fix is closer to context and prompt budgeting than retry accounting. Check which mechanism actually drove the increase before applying either fix.