Token demand
Spends on: What you send the model, and how much you let it write back.
Hides in: A prompt template that gathered context. Nobody re-checked it after it stopped being small.
Watch: p95 tokens per request
Cost
The AI Cost Model: where the spend actually goes.
Six surfaces, 24 named levers, five ways a cost estimate lies, and a scoreboard that keeps them honest. This is the discipline behind the cost stage of the Handbook. It sells no tool.
A per-token price on a pricing page prices one clean call. Production does not run one clean call per outcome. It runs as many attempts as it needs. Retries, fallbacks and chained agent calls never show up in "tokens times price." That gap is why real spend often lands well above the budget. Teams find it on an invoice, after the design review is over.
Cost has six surfaces. Spend hides in the one nobody measures. Skipping a surface does not make it free. The spend still happens. The monthly invoice just attributes it to no one in particular.
The six surfaces
Where the money actually goes.#
Each surface hides spend a different way. None of them replace each other.
Model routing
Spends on: Which model tier ever sees a given request.
Hides in: A feature that defaulted to the frontier model on day one and never got benchmarked against anything cheaper.
Watch: Blended cost per resolved request
Caching and reuse
Spends on: Whatever gets computed, embedded, or generated twice for the same reason.
Hides in: A single volatile token near the front of a prompt, which voids a discount nobody checked against provider usage data.
Watch: Cache-read share of input tokens
Retrieval and the data pipeline
Spends on: Embedding, vector storage, and reindexing, running whether or not anyone reads the result.
Hides in: A finance review that reads this as an infrastructure line item and never connects it to the AI budget at all.
Watch: Tokens of retrieved context per query
Failure economics
Spends on: Every retry, fallback call, and extra step in an agent run that technically finished.
Hides in: A per-token estimate that priced one clean attempt, when production runs however many attempts it actually takes.
Watch: Retry spend as a share of total spend
Attribution and control
Spends on: Nothing directly. This is the instrumentation layer that says which feature spent everything above.
Hides in: A cost spike that shows up as "the AI bill went up" with no feature name attached to it.
Watch: Share of spend with a resolved feature tag
The lever bank
24 levers, six surfaces.#
Filter by stage: design, pre-launch or in production. Then copy the visible list as a Markdown checklist.
Showing all 24 levers
01 Token demand
Every dollar starts here. You pay for what you send the model and for how much you let it write back. A demo prompt is small and hand-trimmed. A production prompt grows. It gathers retrieved chunks, resent conversation history and few-shot examples. Soon nobody remembers why half of it is still there. None of that shows up as its own line on the bill. It shows up as more tokens, priced at the input or output rate.
- What it changes
- The token count of what you send, before the model ever runs.
- How to build it
- Cap retrieved chunk count and re-sent conversation history per turn; summarize or drop old turns instead of resending them verbatim.
- Moves
- Cuts input tokens on chat and RAG workloads that carry full history. The size depends on how much dead weight the prompt held. Measure your own prompt before you trust a percentage.
- Tradeoff
- Trim too hard and the model loses context it needed. Every cut must clear the eval suite before it counts as a saving.
- Prove it
- p95 input tokens per request, before and after.
- What it changes
- How much the model is allowed to generate: the half of the token bill this lever most often gets ignored.
- How to build it
- Set a max-token ceiling matched to the task, and ask for the shortest answer that satisfies it rather than the most complete one.
- Moves
- Output tokens are typically priced well above input tokens, so trimming a needlessly long response can move the bill more than trimming the prompt does.
- Tradeoff
- A cap that is too tight cuts a real answer mid-sentence. A user sees a quality bug long before finance sees a saving.
- Prove it
- p95 output tokens per request, paired with the truncation rate.
- What it changes
- Free-form prose the next step in the pipeline has to parse back out anyway.
- How to build it
- Structured output, a JSON schema or a typed tool return, for anything downstream code consumes instead of a written explanation.
- Moves
- A structured answer is routinely a fraction of the tokens of the prose explanation it replaces, for the same or better downstream accuracy.
- Tradeoff
- An overly narrow schema pressures the model to invent a value for a field it doesn't actually have. That's a quality problem as much as a cost one.
- Prove it
- Output tokens per request, measured before and after the switch from prose to structured output.
- What it changes
- How many few-shot examples ride along on every single call to the feature.
- How to build it
- A/B the example count against the golden-set eval, and keep the smallest count that doesn't move the pass rate.
- Moves
- Every example removed is a fixed per-call saving, multiplied by every request the prompt template ever serves.
- Tradeoff
- Fewer examples can narrow the range of cases the model handles well. This holds only if the eval set covers what those examples taught.
- Prove it
- Few-shot token count in the template, checked against golden-set pass rate.
02 Model routing
Not every request needs the strongest model on the price list. Many teams pay frontier rates for a classification task that a small model would get right. That is the easiest saving they leave unclaimed. This surface covers which model tier handles each request. Decide it per request class, not once for the whole feature. Then check one thing: when a cheaper option clears the bar, is the switch a config change or a rewrite?
Run a cheap-first model cascade
Pre-launch- What it changes
- Which model tier handles a given request, decided per request instead of once for the whole feature.
- How to build it
- Route every request to the cheapest model first, validate the attempt against a schema or confidence check, and escalate only on a fail.
- Moves
- Cuts blended spend on request classes where most traffic is easy. One fintech engagement saw a 41% blended drop after cutover; that's a result from one engagement, not a number to assume for yours.
- Tradeoff
- The saving is real only if the escalation signal is calibrated. A lenient confidence check escalates almost nothing and under-serves the hard tail.
- Prove it
- Escalation rate, tracked alongside blended cost per resolved request.
- What it changes
- Whether a pricing change or a new model release is a config edit or a rewrite.
- How to build it
- One internal interface every call site goes through; the routing logic, not the calling code, picks the provider and model.
- Moves
- Doesn't cut spend by itself. It's what makes every other routing lever on this page cheap to keep adjusting instead of a migration project each time.
- Tradeoff
- Building the abstraction layer is real upfront engineering cost, paid before any of the routing savings arrive.
- Prove it
- Time to add or swap a model, measured honestly the first time it actually happens.
- What it changes
- The model tier a new feature launches with, before any cascade or routing logic exists at all.
- How to build it
- Benchmark the cheapest model that clears the feature's own eval bar before writing the routing layer, not after.
- Moves
- A feature that ships with a needlessly strong default can cost a multiple of a properly sized one. It does so until someone notices.
- Tradeoff
- A cheaper default must still clear the eval bar. An honest benchmark takes time before day one, not after a complaint.
- Prove it
- Eval pass rate at the chosen default tier, compared against the next tier up.
- What it changes
- Whether a request pays a per-token API rate at all, or a fixed infrastructure cost instead.
- How to build it
- For one narrow, well-defined, high-volume task, benchmark a self-hosted open-weight model against the same eval suite as the API option.
- Moves
- Can flip the economics from variable-per-call to fixed-per-month at high volume. The crossover point depends on your own traffic. Model it. Do not assume it.
- Tradeoff
- Trades a per-token bill for GPU-fleet and MLOps ownership, a real cost this page will not pretend away.
- Prove it
- Modeled break-even volume, checked against actual traffic.
03 Caching and reuse
You do not get a cache discount once. You must earn it again on every call. One volatile token near the front of a prompt voids it for the whole request. The system gives no error and no warning. This surface covers everything that stops you paying twice for the same computation. That includes prompt caching, retrieval caching and reuse of an answer in place of a new generation.
- What it changes
- Whether a request's shared prefix is byte-identical to the last one, the only thing that makes a cache discount apply at all.
- How to build it
- Static instructions and tool definitions first, in a fixed serialization order; anything that changes per request goes last.
- Moves
- The provider sets the discount rate. A broken cache means you pay full price on tokens that should have been discounted, on every call.
- Tradeoff
- The discipline must hold across every template edit. One volatile token near the front reopens the whole leak without warning.
- Prove it
- Cache-read share of input tokens, read straight from provider usage data.
- What it changes
- Whether the same lookup runs, and bills, twice.
- How to build it
- A cache keyed on the normalized query or document hash, checked before the retrieval or embedding call runs again.
- Moves
- On workloads with repeated queries, each hit removes the cost of a full retrieval-and-embedding round trip.
- Tradeoff
- Cache invalidation on a changed source document is the freshness problem retrieval already has. Get it wrong and the answer goes stale.
- Prove it
- Retrieval cache-hit rate, paired with the index-staleness lag it must not be allowed to worsen.
Memoize identical or near-identical requests
Pre-launch- What it changes
- Whether an identical request pays for a fresh generation every time it arrives.
- How to build it
- Hash the normalized request and check a short-lived response cache before calling the model at all.
- Moves
- On features with many repeated requests, this can remove the call for a share of traffic. An FAQ-shaped assistant and a repeated classification are examples.
- Tradeoff
- Only safe where a slightly stale answer is fine. Anything time-sensitive or personalized needs a short TTL, or no cache.
- Prove it
- Memoization hit rate, paired with the TTL it runs on.
Measure the cache-read share, don't assume it
In production- What it changes
- Whether "we ordered the prompt correctly" is a measured fact or a belief nobody has re-checked since launch.
- How to build it
- Pull cache-read versus total input tokens from provider usage data on a schedule, not once at launch and never again.
- Moves
- Does not cut cost on its own. It turns a broken cache into a caught regression, not a permanent, invisible leak.
- Tradeoff
- Needs a dashboard and a person who looks at it. A number nobody checks might as well not be measured.
- Prove it
- Cache-read share trend, alerted on a drop.
04 Retrieval and the data pipeline
Teams rarely count retrieval spend as AI spend. It looks like a database and infrastructure bill, not a model bill. Vector storage, embedding calls and reindex jobs run on a schedule. They run whether or not anyone reads what they produce. Most cost conversations skip this surface. Then someone asks why the vector-store invoice keeps climbing.
- What it changes
- How much gets embedded, stored, and re-retrieved for every single query.
- How to build it
- Tune chunk size and retrieval top-k against the eval suite's recall@k, not whatever a library ships as its default.
- Moves
- Oversized chunks or a high top-k inflate the vector-store bill. They also inflate the tokens spent on context the generator never needed.
- Tradeoff
- Cut too far and recall drops. This lever tunes cost and quality together. It is not a pure cost lever.
- Prove it
- Recall@k, paired with tokens of retrieved context per query.
- What it changes
- How often, and how much of, the pipeline reprocesses content that hasn't actually changed.
- How to build it
- Change-data-capture ingest instead of a full periodic recrawl, with tombstones for anything deleted at the source.
- Moves
- A full recrawl reprocesses and re-embeds everything, changed or not. A change-driven pipeline pays only for what changed.
- Tradeoff
- Requires a real change feed from the source system, which not every source exposes cleanly.
- Prove it
- Index-staleness lag, paired with compute spent per ingest run.
- What it changes
- How many candidate documents get pulled, and paid for, before the generator ever sees them.
- How to build it
- A cheap first-pass retrieval over a smaller candidate set, then a reranking step that picks the best few.
- Moves
- Cuts the volume of retrieved context that reaches the generator, compared with a brute-force high-top-k fetch. It keeps the precision that fetch sought.
- Tradeoff
- The rerank step is a call or a lightweight model. Count it in the total. Do not assume it is free.
- Prove it
- Tokens of retrieved context per query, paired with groundedness rate.
Retire sources nobody queries
In production- What it changes
- Whether every ingested source is still earning its ongoing embedding and storage cost.
- How to build it
- Track query volume per source, and retire or de-prioritize the ones with none, on a real recurring schedule.
- Moves
- An abandoned source costs money on every reindex long after anyone reads it. The saving grows with the time since the last audit of the source list.
- Tradeoff
- Needs an owner for the audit. It is easy to defer forever, because nothing breaks while it waits.
- Prove it
- Query volume per source, checked against ongoing ingest spend per source.
05 Failure economics
A retry on a malformed response is a billed event. So is a fallback to a pricier model, and so is an extra step in an agent loop. A per-token estimate prices none of them. The estimate assumes one clean attempt per outcome. Production runs as many attempts as it needs. Most of the gap between the sticker price and the real invoice comes from this surface.
Budget cost per completed task, not per call
Pre-launch- What it changes
- What "acceptable spend" means for a feature, inclusive of every retry and fallback it actually took to get there.
- How to build it
- Divide total spend for a feature by completed-task count, not call count, and alert on that ratio as well as the total.
- Moves
- This lever measures. It does not cut. It makes retry-amplified spend visible before it surprises you on the invoice.
- Tradeoff
- Requires wiring a real completion event through the system, which most teams have not instrumented.
- Prove it
- Cost per completed task, tracked over time alongside total spend.
- What it changes
- How many calls a single task is allowed to make before the harness forces it to a terminal state.
- How to build it
- A bounded-agent-loop budget that the harness enforces on step count, tokens and wall-clock time. The model never decides when to stop.
- Moves
- An unbounded loop can multiply the cost of one task several times. Nobody notices until it fails to end cleanly.
- Tradeoff
- A ceiling set too low cuts off a task that needed a few more steps. Tune it against real task-length data. Do not guess.
- Prove it
- Terminal-state rate and step-count distribution.
Cap retry and fallback depth explicitly
Pre-launch- What it changes
- Whether a rate-limit or malformed-response error triggers one bounded retry, or a silent, unbounded chain of them.
- How to build it
- A fixed retry ceiling with backoff. Add a separate fallback path with its own cap. Count it apart from ordinary retries.
- Moves
- Removes the pattern where a retry loop multiplies the bill on the requests that already struggle.
- Tradeoff
- A ceiling that is too low turns a transient blip into a failure, not a recovery. This tradeoff covers reliability as much as cost.
- Prove it
- Retry rate, and retry spend as a share of total spend, tracked separately from clean-call spend.
- What it changes
- Whether a retry of a partly successful call creates a duplicate side effect and a duplicate bill, or a safe no-op.
- How to build it
- An idempotency key on every write-scoped tool call, checked before the action runs a second time.
- Moves
- This lever is about correctness cost more than token cost. A duplicated write downstream can cost far more than the extra API call that caused it.
- Tradeoff
- Requires the downstream system to actually support an idempotency key, which not every legacy API does.
- Prove it
- Duplicate-write rate on retried calls, aiming for zero.
06 Attribution and control
The five surfaces above do not help if nobody can trace a cost spike to its feature. They do not help if nobody can stop that feature without waiting for a deploy window. This surface is the layer under every other one. It holds tagging, budgets, alerts and kill switches. With it, you find a regression the same afternoon. Without it, you find it on the monthly invoice.
- What it changes
- Whether a cost spike can be traced to the feature responsible, or only to "AI spend" as one undifferentiated total.
- How to build it
- A feature and request-class tag on every logged call, joined to billing data at the same granularity.
- Moves
- Turns "the AI bill went up" into "feature X's retry rate went up." Only the second sentence is one anyone can act on.
- Tradeoff
- Enforce tagging at every call site. If you do not, the untagged share grows into its own unattributed bucket.
- Prove it
- Share of spend with a resolved feature tag, aiming for the high end.
Give each feature its own budget and alert threshold
In production- What it changes
- Whether a new cost regression pages someone the hour it starts, or surfaces on next month's invoice.
- How to build it
- A per-feature spend ceiling with an alert threshold well under it, checked against actual spend on a short interval.
- Moves
- Converts a runaway feature into a same-day alert. Without it, weeks pass between the start of the regression and the first notice.
- Tradeoff
- Needs per-feature spend data in close to real time. The tagging lever above is a prerequisite.
- Prove it
- Time between the start of a spend regression and the alert. Measure it against a real incident, not a drill.
Give every feature its own kill switch
Pre-launch- What it changes
- Whether stopping one feature's spend requires a deploy, or a flag flip.
- How to build it
- A feature flag or rate limit that can throttle or disable one AI-backed feature without taking the rest of the product down with it.
- Moves
- Turns a runaway feature from an incident that lasts until the next deploy window into one that stops within minutes.
- Tradeoff
- Wire the flag into the call path ahead of time. A retrofit in the middle of an incident is too late.
- Prove it
- Time to throttle a feature, tested as a drill rather than assumed to work.
Reconcile the model against the invoice, on a schedule
In production- What it changes
- Whether the gap between the modeled estimate and the real bill is caught in a week, or in a quarter.
- How to build it
- A recurring reconciliation between the modeled per-outcome cost and actual billing data. A named person owns it.
- Moves
- This lever catches stale assumptions in every other lever. Examples are a cache that stopped hitting, a cascade whose escalation rate drifted, and a source that someone re-added.
- Tradeoff
- Someone must own this on a calendar. If nobody does, it becomes a surprise on the invoice again. This page opened with that failure.
- Prove it
- Modeled-vs-actual variance, tracked per reconciliation cycle.
No levers match that combination. Clear a filter to see more.
Five ways an estimate lies
A cost model is a claim, not a fact.#
Every one of these produces a confidently wrong number, not an obviously broken one.
Sticker-price math
Looks like: A cost estimate multiplies a single blended $-per-token rate by a token count, as if input and output tokens cost the same.
Costs you: The estimate is wrong in a predictable direction, usually low. Output tokens are typically priced well above input tokens. A chatty feature spends most of its budget on the more expensive half.
Fix: Price the input and output halves of every request separately, each at its own rate. Then add them together.
See the lever: Bound output length on purpose →Happy-path volume
Looks like: A volume forecast counts real user requests only: no retries, no dev and staging traffic, no eval-suite runs against the same model.
Costs you: Every one of those omitted categories is a billed call. A retry-heavy feature can run several billed attempts per completed task, even in a healthy system. The happy-path forecast counted none of them.
Fix: Model cost per completed task, including every retry and fallback it took. Do not model cost per request on the assumption that the first attempt lands.
See the lever: Cap retry and fallback depth explicitly →An assumed cache-hit rate
Looks like: The estimate applies a caching discount because the prompt "should" cache well. Nobody pulled the cache-read number from provider usage data.
Costs you: A single volatile token near the front of the prompt voids the discount. The system reports no error when that happens. The estimate keeps assuming a saving that stopped weeks ago.
Fix: Read the real cache-read share from provider usage data before you build it into a forecast. Re-check it on a schedule.
See the lever: Measure the cache-read share, don't assume it →Cost per request instead of cost per outcome
Looks like: The reported figure is spend divided by API calls made. The business pays for resolved outcomes, and cares about them.
Costs you: A feature with a low failure rate and a feature that retries three times per success can report the same cost-per-call number. The second feature costs three times as much per outcome.
Fix: Divide total spend by completed-task count, not call count. Track the ratio between the two over time.
See the lever: Budget cost per completed task, not per call →A pilot curve extrapolated linearly
Looks like: A cost projection multiplies the pilot's per-user cost by the target user count. It assumes cost scales in a straight line with volume.
Costs you: Context growth and agent fan-out tend to grow faster than user count. A chat feature that resends growing history costs more per user at scale than the pilot showed. So does an agent workflow that spawns more sub-tasks under real load.
Fix: Model the growth curve for context size and step count apart from the curve for user count. Extrapolate each on its own.
See the lever: Give every agent run a hard ceiling → The scoreboard
Six numbers, and how each one lies.#
Every one of these is measurable today. None of them is trustworthy read alone. Pair it with the number next to it.
Cost per resolved outcome
- Why it matters
- What the business pays for one completed task. It is not the price of one API call.
- How it misleads
- On its own it says nothing about whether outcomes land. A feature can look cheap per outcome because it resolves so rarely that the denominator stays small.
- Pair with
- Resolution rate
Cost per 1,000 requests
- Why it matters
- A volume-normalized number. Finance can forecast it against committed traffic, whatever the size of any one request.
- How it misleads
- A stable per-1,000-requests number can hide a heavier request shape. The request count stays the same, but each request carries more tokens.
- Pair with
- p95 tokens per request
Cache-hit rate
- Why it matters
- Says whether the caching discipline this page argues for works, not only whether you built it.
- How it misleads
- A healthy-looking hit rate can still cover a small share of total spend. This happens when the cached calls were cheap and the uncached calls hold the real money.
- Pair with
- Share of spend actually discounted
Retry rate
- Why it matters
- The leading indicator for retry-amplified spend, visible before the invoice confirms it.
- How it misleads
- A feature can have a low retry rate and a few very expensive retries. Most of its marginal spend can still be retries, even though the rate looks fine.
- Pair with
- Retry spend as a share of total spend
Monthly run-rate
- Why it matters
- The number finance budgets against. Update it as real traffic replaces the pilot estimate.
- How it misleads
- A flat run-rate over a few months can mask a volume increase offset by a routing improvement. Together they look like no change, until one of them stops.
- Pair with
- Forecast at committed volume
Inference spend
- Why it matters
- The line item that procurement and finance recognize first. It is also the one a per-token pricing page prices.
- How it misleads
- It excludes retrieval infrastructure, evals, observability, and the engineering time spent operating all of it. A system can look cheap on inference alone and expensive in total.
- Pair with
- Total cost of ownership, evals and infra and engineering included
What this is built from
Verifiable, not claimed.#
No invented benchmark numbers, no vendor price list. Just what's already documented on this site.
- The benchmark behind this page models cost per successful outcome across four common workload shapes. It states every price, token count and multiplier. You can rederive each figure against your own traffic. Insight: AI cost benchmark by workload
- This site covered the gap between a per-token sticker price and a production invoice before this page existed. That piece explains why the gap opens. The levers above close it. Insight: What an AI feature actually costs in production
- This site already documents model cascade and stable-prefix prompt caching in full. This page connects them and the other levers above into one argument. Each is no longer a standalone technique. Pattern: Model cascade
- Retry-amplified spend and prompt-cache-never-hits are named failure modes on this site. Each one is the reason a specific lever above exists. Failure mode: Retry-amplified spend
- A published engagement cut blended cost per resolved request by 41% with a model-cascade routing layer. That is one client's number, shown with its own baseline and window. Do not expect it elsewhere. Case study: Cutting inference spend without one vendor
- The AI Cost Calculator models the multiplier this page argues for. It applies retries, context growth and eval overhead to your own volume. It does not use a vendor sticker price. AI Cost Calculator
- CostMon exists because the studio hit its own version of the attribution-and-control surface above. It ran AI, cloud and SaaS spend for nine products across a dozen billing consoles. No console showed a shared total. CostMon
- FreeTier answers the surface-zero question before any of these levers apply. Is the workload already free somewhere? Ask that before you build the paid alternative. FreeTier
Sources
- FinOps Foundation - FinOps Framework Overview
The cost-management framework our cost model's phases map onto. Retrieved 2026-08-24.
- FinOps Foundation - FinOps Capabilities
The capability list our cost levers are checked against. Retrieved 2026-08-24.
Not sure where your spend goes
A Ship Audit runs this model against your own system. It tells you which surface costs you money. It also shows which surfaces only look incomplete on paper.
Questions
Before you trust the estimate.#
What teams ask us before they model AI spend.
01 Why is our inference bill so much higher than what we estimated?
The estimate almost certainly priced one clean call per outcome. Production runs retries and fallbacks. For agents, it runs several chained calls per completed task. A simple tokens-times-price model shows none of them. See the five ways an estimate lies above. The gap is usually more than one of them at once.
Link to this answer: Why is our inference bill so much higher than what we estimated?02 What's the single fastest lever to pull first?
Measure cost per completed task, not cost per call. It costs nothing to build beyond a completion event. It also makes every other cost problem on this page visible, so nobody has to assume.
Link to this answer: What's the single fastest lever to pull first?03 Do we need all six surfaces on day one?
No. Start with token demand and model routing. Every feature has those two surfaces, whatever its shape. Add caching once volume is real. Add retrieval-pipeline cost once there is a knowledge base to maintain. Add failure economics and attribution once there is enough traffic for a regression to cost money.
Link to this answer: Do we need all six surfaces on day one?04 Isn't cutting tokens just going to hurt quality?
It can. That is why every lever on this page is checked against the eval suite. The invoice alone is not enough to ship a cut. A cut that drops the golden-set pass rate is a quality regression with a smaller bill. It is not a cost win.
Link to this answer: Isn't cutting tokens just going to hurt quality?05 How is this different from the AI Cost Calculator?
The calculator turns your own volume and failure-rate assumptions into a number. This page covers the levers that move that number once you have it. It lists six surfaces and 24 named changes. It also lists five ways the estimate behind the calculator can be wrong before you touch a lever.
Link to this answer: How is this different from the AI Cost Calculator?06 Why no per-token price table?
Per-token prices change every few weeks. A table of them would be the fastest-rotting page on this site within a quarter. This page uses ratios, shapes and levers. They stay true whatever a provider charges this month. For the arithmetic, use the calculator.
Link to this answer: Why no per-token price table?