# The AI Cost Model: where the spend actually goes.

Source: https://customlabs.io/cost/
Updated: 2026-09-20

Cost

# The AI Cost Model: where the spend actually goes.

Six surfaces, 24 named levers, five ways a cost estimate lies, and the scoreboard that keeps them honest. This is the discipline behind the cost stage of the Handbook, not a pitch for a tool.

Updated September 20, 2026 · First published August 11, 2026

·

28 min read

·

Key takeaways

- → Failure economics counts the cost of retries and fallbacks as their own line item.
- → An assumed cache-hit rate is named as one of five ways a cost estimate is already wrong.
- → Cost per resolved outcome is what the business pays for one completed task, not one call.
- → Inference spend excludes retrieval infrastructure and the [evals](https://customlabs.io/glossary/eval-suite/) pipeline around it.
- → 24 levers span six surfaces, from token demand to attribution and control.
- → Monthly run-rate is the number finance actually budgets against, not the pilot estimate.

A per-token price on a pricing page prices one clean call. Production doesn't run one clean call per outcome. It runs however many attempts it actually takes, with retries, fallbacks, and chained agent calls that never show up in "tokens times price." That gap is why real spend routinely lands well above the number that got budgeted, discovered for the first time on an invoice instead of in a design review.

Cost isn't one number to watch. It's six surfaces, and spend hides in whichever one nobody is currently measuring. Skipping a surface doesn't make it free. It means the spend still happens, just somewhere the monthly invoice attributes to no one in particular.

The six surfaces

## Where the money actually goes.

Each surface hides spend a different way. None of them replace each other.

Each of the six cost surfaces sits in the stage where most of its levers apply. Four land at design; one at pre-launch; and one in production.

01

### Token demand

**Spends on:** What you send the model, and how much you let it write back.

**Hides in:** A prompt template that accumulated context nobody re-checked once it stopped being small.

**Watch:** p95 tokens per request

02

### Model routing

**Spends on:** Which model tier ever sees a given request.

**Hides in:** A feature that defaulted to the frontier model on day one and never got benchmarked against anything cheaper.

**Watch:** Blended cost per resolved request

03

### Caching and reuse

**Spends on:** Whatever gets computed, embedded, or generated twice for the same reason.

**Hides in:** A single volatile token near the front of a prompt, quietly voiding a discount nobody's checked against provider usage data.

**Watch:** Cache-read share of input tokens

04

### Retrieval and the data pipeline

**Spends on:** Embedding, vector storage, and reindexing, running whether or not anyone reads the result.

**Hides in:** A finance review that reads this as an infrastructure line item and never connects it to the AI budget at all.

**Watch:** Tokens of retrieved context per query

05

### Failure economics

**Spends on:** Every retry, fallback call, and extra step in an agent run that technically finished.

**Hides in:** A per-token estimate that priced one clean attempt, when production runs however many attempts it actually takes.

**Watch:** Retry spend as a share of total spend

06

### Attribution and control

**Spends on:** Nothing directly. This is the instrumentation layer that says which feature spent everything above.

**Hides in:** A cost spike that shows up as "the AI bill went up" with no feature name attached to it.

**Watch:** Share of spend with a resolved feature tag

The lever bank

## 24 levers, six surfaces.

Filter by when it applies (design, pre-launch, in production), then copy the visible list as a Markdown checklist.

Design Pre-launch In production

Showing all 24 levers

Copy as Markdown

### 01 Token demand

Every dollar starts here: what you send the model and how much you let it generate in return. A demo prompt is small and hand-trimmed; a production prompt accumulates retrieved chunks, resent conversation history, and few-shot examples until nobody remembers why half of it is still there. None of that shows up as its own line item on the bill. It just shows up as more tokens, priced at whichever rate the input or output half happens to carry.

#### Trim input to what the task needs

Design

**What it changes**
The token count of what you send, before the model ever runs.

**How to build it**
Cap retrieved chunk count and re-sent conversation history per turn; summarize or drop old turns instead of resending them verbatim.

**Moves**
Cuts input tokens meaningfully on chat and RAG workloads carrying full history. The exact size depends entirely on how much dead weight was in there to begin with, so measure your own prompt before trusting a percentage.

**Tradeoff**
Trim too hard and the model loses context it actually needed; every cut has to clear the eval suite before it counts as a saving.

**Prove it**
p95 input tokens per request, before and after.

[Glossary: Token](https://customlabs.io/glossary/token/)[Glossary: Context window](https://customlabs.io/glossary/context-window/)

#### Bound output length on purpose

Design

**What it changes**
How much the model is allowed to generate: the half of the token bill this lever most often gets ignored.

**How to build it**
Set a max-token ceiling matched to the task, and ask for the shortest answer that satisfies it rather than the most complete one.

**Moves**
Output tokens are typically priced well above input tokens, so trimming a needlessly long response can move the bill more than trimming the prompt does.

**Tradeoff**
Too tight a cap truncates a real answer mid-sentence, which reads as a quality bug to a user long before it reads as a savings to finance.

**Prove it**
p95 output tokens per request, paired with the truncation rate.

#### Ask for the field, not the paragraph

Design

**What it changes**
Free-form prose the next step in the pipeline has to parse back out anyway.

**How to build it**
Structured output, a JSON schema or a typed tool return, for anything downstream code consumes instead of a written explanation.

**Moves**
A structured answer is routinely a fraction of the tokens of the prose explanation it replaces, for the same or better downstream accuracy.

**Tradeoff**
An overly narrow schema pressures the model to invent a value for a field it doesn't actually have. That's a quality problem as much as a cost one.

**Prove it**
Output tokens per request, measured before and after the switch from prose to structured output.

[Glossary: Structured output](https://customlabs.io/glossary/structured-output/)

#### Cut the few-shot count to the smallest set that holds

Design

**What it changes**
How many few-shot examples ride along on every single call to the feature.

**How to build it**
A/B the example count against the golden-set eval, and keep the smallest count that doesn't move the pass rate.

**Moves**
Every example removed is a fixed per-call saving, multiplied by every request the prompt template ever serves.

**Tradeoff**
Fewer examples can quietly narrow the range of cases the model handles well. This only holds if the eval set actually covers what those examples were teaching.

**Prove it**
Few-shot token count in the template, checked against golden-set pass rate.

[Insight: Evals before you ship](https://customlabs.io/insights/evals-before-you-ship/)

### 02 Model routing

Not every request needs the strongest model on the price list, and paying frontier rates for a classification task a small model would have gotten right is the easiest efficiency most teams leave on the table. This surface is about which model tier handles a given request, decided per request class instead of once for the whole feature, and whether that decision is a config change or a rewrite the next time a cheaper option clears the bar.

#### Run a cheap-first model cascade

Pre-launch

**What it changes**
Which model tier handles a given request, decided per request instead of once for the whole feature.

**How to build it**
Route every request to the cheapest model first, validate the attempt against a schema or confidence check, and escalate only on a fail.

**Moves**
Meaningfully cuts blended spend on request classes where most traffic is genuinely easy. One fintech engagement saw a 41% blended drop after cutover; that's a result from one engagement, not a number to assume for yours.

**Tradeoff**
The saving is only real if the escalation signal is actually calibrated. A lenient confidence check escalates almost nothing and quietly under-serves the hard tail.

**Prove it**
Escalation rate, tracked alongside blended cost per resolved request.

[Pattern: Model cascade](https://customlabs.io/patterns/model-cascade/)[Case study: Cutting inference spend without one vendor](https://customlabs.io/case-studies/model-agnostic-routing-cut-inference-spend/)

#### Put routing behind one internal interface

Design

**What it changes**
Whether a pricing change or a new model release is a config edit or a rewrite.

**How to build it**
One internal interface every call site goes through; the routing logic, not the calling code, picks the provider and model.

**Moves**
Doesn't cut spend by itself. It's what makes every other routing lever on this page cheap to keep adjusting instead of a migration project each time.

**Tradeoff**
Building the abstraction layer is real upfront engineering cost, paid before any of the routing savings arrive.

**Prove it**
Time to add or swap a model, measured honestly the first time it actually happens.

[Insight: Model-agnostic by design](https://customlabs.io/insights/model-agnostic-by-design/)

#### Benchmark the default before you ship it

Design

**What it changes**
The model tier a new feature launches with, before any cascade or routing logic exists at all.

**How to build it**
Benchmark the cheapest model that clears the feature's own eval bar before writing the routing layer, not after.

**Moves**
A feature that ships against a needlessly strong default can run a real multiple over a properly sized one, for however long it takes anyone to notice.

**Tradeoff**
A cheaper default still has to clear the eval bar, and benchmarking it honestly takes real time before day one, not after a complaint.

**Prove it**
Eval pass rate at the chosen default tier, compared against the next tier up.

[Glossary: Inference cost](https://customlabs.io/glossary/inference-cost/)[Insight: AI cost benchmark by workload](https://customlabs.io/insights/ai-cost-benchmark-by-workload/)

#### Self-host for a single, high-volume, stable task

Design

**What it changes**
Whether a request pays a per-token API rate at all, or a fixed infrastructure cost instead.

**How to build it**
For one narrow, well-defined, high-volume task, benchmark a self-hosted open-weight model against the same eval suite as the API option.

**Moves**
Can flip the economics from variable-per-call to fixed-per-month at high enough volume. The crossover point depends entirely on your own traffic and has to be modeled, not assumed.

**Tradeoff**
Trades a per-token bill for GPU-fleet and MLOps ownership, a real cost this page will not pretend away.

**Prove it**
Modeled break-even volume, checked against actual traffic.

[Compare: Open-weight vs. frontier API](https://customlabs.io/compare/open-weight-vs-frontier-api/)

### 03 Caching and reuse

A cache discount isn't something you get once, it's something you have to keep earning on every single call. One volatile token near the front of a prompt is enough to void it for the entire request, with no error and no warning that anything changed. This surface covers everything that stops the same computation from being paid for twice: prompt caching, retrieval caching, and reusing an answer instead of regenerating it.

#### Order the prompt invariant-first

Design

**What it changes**
Whether a request's shared prefix is byte-identical to the last one, the only thing that makes a cache discount apply at all.

**How to build it**
Static instructions and tool definitions first, in a fixed serialization order; anything that changes per request goes last.

**Moves**
The discount rate is set by the provider, but a broken cache means paying full price on tokens that should have been discounted, on every single call.

**Tradeoff**
The discipline has to hold across every template edit. One volatile token reintroduced near the front silently reopens the whole leak.

**Prove it**
Cache-read share of input tokens, read straight from provider usage data.

[Pattern: Stable-prefix prompt caching](https://customlabs.io/patterns/stable-prefix-prompt-caching/)[Failure mode: Prompt cache never hits](https://customlabs.io/failure-modes/prompt-cache-never-hits/)

#### Cache repeated retrieval and embedding lookups

Pre-launch

**What it changes**
Whether the same lookup runs, and bills, twice.

**How to build it**
A cache keyed on the normalized query or document hash, checked before the retrieval or embedding call runs again.

**Moves**
On workloads with real query repetition, this removes a full retrieval-and-embedding round trip's cost each time it hits.

**Tradeoff**
Cache invalidation on a changed source document is the same freshness problem retrieval already has. Get it wrong and the answer goes stale along with the price.

**Prove it**
Retrieval cache-hit rate, paired with the index-staleness lag it must not be allowed to worsen.

[Failure mode: Stale index serves deleted content](https://customlabs.io/failure-modes/stale-index-serves-deleted-content/)

#### Memoize identical or near-identical requests

Pre-launch

**What it changes**
Whether an identical request pays for a fresh generation every time it arrives.

**How to build it**
Hash the normalized request and check a short-lived response cache before calling the model at all.

**Moves**
On features with real repeat-request volume (an FAQ-shaped assistant, a repeated classification), this can remove the call entirely for a meaningful share of traffic.

**Tradeoff**
Only safe where a slightly stale answer is genuinely fine; anything time-sensitive or personalized needs a short TTL, or no cache at all.

**Prove it**
Memoization hit rate, paired with the TTL it runs on.

#### Measure the cache-read share, don't assume it

In production

**What it changes**
Whether "we ordered the prompt correctly" is a measured fact or a belief nobody has re-checked since launch.

**How to build it**
Pull cache-read versus total input tokens from provider usage data on a schedule, not once at launch and never again.

**Moves**
Doesn't cut cost on its own. It turns a silently broken cache into a caught regression instead of a permanent, invisible leak.

**Tradeoff**
Needs a dashboard and a person who actually looks at it; a number nobody checks might as well not be measured.

**Prove it**
Cache-read share trend, alerted on a drop.

[Failure mode: Prompt cache never hits](https://customlabs.io/failure-modes/prompt-cache-never-hits/)

### 04 Retrieval and the data pipeline

Retrieval spend rarely gets counted as AI spend, because it looks like a database and infrastructure bill rather than a model one. Vector storage, embedding calls, and reindex jobs run on a schedule whether or not anyone ever reads what they produce. This is the surface most cost conversations skip entirely, right up until someone asks why the vector-store invoice keeps climbing on its own.

#### Size chunks and top-k against recall, not a default

Design

**What it changes**
How much gets embedded, stored, and re-retrieved for every single query.

**How to build it**
Tune chunk size and retrieval top-k against the eval suite's recall@k, not whatever a library ships as its default.

**Moves**
Oversized chunks or an unnecessarily high top-k inflate both the vector-store bill and the tokens spent on context the generator never needed.

**Tradeoff**
Cut too aggressively and recall drops. This is a joint cost-and-quality tuning problem, not a pure cost lever.

**Prove it**
Recall@k, paired with tokens of retrieved context per query.

[Pattern: Structure-aware chunking](https://customlabs.io/patterns/structure-aware-chunking/)

#### Ingest on change events, not a timer

Design

**What it changes**
How often, and how much of, the pipeline reprocesses content that hasn't actually changed.

**How to build it**
Change-data-capture ingest instead of a full periodic recrawl, with tombstones for anything deleted at the source.

**Moves**
A full recrawl reprocesses and re-embeds everything, changed or not; a change-driven pipeline only ever pays for what actually changed.

**Tradeoff**
Requires a real change feed from the source system, which not every source exposes cleanly.

**Prove it**
Index-staleness lag, paired with compute spent per ingest run.

[Pattern: Change-data-capture ingest](https://customlabs.io/patterns/change-data-capture-ingest/)

#### Fetch fewer candidates, then rerank

Design

**What it changes**
How many candidate documents get pulled, and paid for, before the generator ever sees them.

**How to build it**
A cheap first-pass retrieval over a smaller candidate set, then a reranking step that picks the best few.

**Moves**
Cuts the volume of retrieved context handed to the generator versus a brute-force high-top-k fetch, without giving up the precision that fetch was chasing.

**Tradeoff**
The rerank step is itself a call, or a lightweight model, that has to be counted in the total rather than assumed free.

**Prove it**
Tokens of retrieved context per query, paired with groundedness rate.

[Pattern: Retrieve then rerank](https://customlabs.io/patterns/retrieve-then-rerank/)

#### Retire sources nobody queries

In production

**What it changes**
Whether every ingested source is still earning its ongoing embedding and storage cost.

**How to build it**
Track query volume per source, and retire or de-prioritize the ones with none, on a real recurring schedule.

**Moves**
An abandoned source keeps costing money on every reindex long after anyone reads from it. The saving scales with how long it's been since anyone audited the source list.

**Tradeoff**
Needs a real owner for the audit; it is easy to defer indefinitely because nothing breaks while it sits there.

**Prove it**
Query volume per source, checked against ongoing ingest spend per source.

### 05 Failure economics

A retry on a malformed response, a fallback to a pricier model when the first one fails, and an extra step in an agent loop are all real, billed events that a per-token estimate never priced, because the estimate assumed one clean attempt per outcome and production runs however many attempts it actually takes. This surface is where most of the gap between the sticker price and the real invoice comes from.

#### Budget cost per completed task, not per call

Pre-launch

**What it changes**
What "acceptable spend" means for a feature, inclusive of every retry and fallback it actually took to get there.

**How to build it**
Divide total spend for a feature by completed-task count, not call count, and alert on that ratio as well as the total.

**Moves**
This is a measurement lever, not a cutting one. It makes retry-amplified spend visible before it's a surprise on the invoice.

**Tradeoff**
Requires wiring a real completion event through the system, which most teams have not instrumented.

**Prove it**
Cost per completed task, tracked over time alongside total spend.

[Failure mode: Retry-amplified spend](https://customlabs.io/failure-modes/retry-amplified-spend/)

#### Give every agent run a hard ceiling

Design

**What it changes**
How many calls a single task is allowed to make before the harness forces it to a terminal state.

**How to build it**
A bounded-agent-loop budget enforced by the harness on step count, tokens, and wall-clock time, never left to the model to decide when to stop.

**Moves**
An unbounded loop can multiply a single task's cost several times over before anyone notices it never terminated cleanly.

**Tradeoff**
A ceiling set too low cuts off a task that genuinely needed a few more steps, so it has to be tuned against real task-length data, not guessed.

**Prove it**
Terminal-state rate and step-count distribution.

[Pattern: Bounded agent loop](https://customlabs.io/patterns/bounded-agent-loop/)[Failure mode: Unbounded agent loop](https://customlabs.io/failure-modes/unbounded-agent-loop/)

#### Cap retry and fallback depth explicitly

Pre-launch

**What it changes**
Whether a rate-limit or malformed-response error triggers one bounded retry, or a silent, unbounded chain of them.

**How to build it**
A fixed retry ceiling with backoff, and a separate, also-capped fallback path, counted apart from ordinary retries rather than folded into the same number.

**Moves**
Removes the specific pattern where a retry loop quietly multiplies the bill on exactly the requests that were already struggling.

**Tradeoff**
Too low a ceiling turns a transient blip into an outright failure instead of a recovery. This is a reliability tradeoff as much as a cost one.

**Prove it**
Retry rate, and retry spend as a share of total spend, tracked separately from clean-call spend.

#### Make retried writes idempotent by design

Design

**What it changes**
Whether retrying a call that partially succeeded creates a duplicate side effect, and a duplicate bill, instead of a safe no-op.

**How to build it**
An idempotency key on every write-scoped tool call, checked before the action runs a second time.

**Moves**
This is about correctness cost more than token cost. A duplicated write downstream can cost far more than the extra API call that caused it.

**Tradeoff**
Requires the downstream system to actually support an idempotency key, which not every legacy API does.

**Prove it**
Duplicate-write rate on retried calls, aiming for zero.

[Glossary: Idempotency](https://customlabs.io/glossary/idempotency/)

### 06 Attribution and control

None of the five surfaces above matter if nobody can trace a cost spike back to the feature that caused it, or stop that feature without waiting for a deploy window. This surface is the tagging, budgeting, alerting, and kill-switch layer underneath every other one: the difference between finding a regression on the monthly invoice and finding it the same afternoon.

#### Tag every call with the feature that caused it

Design

**What it changes**
Whether a cost spike can be traced to the feature responsible, or only to "AI spend" as one undifferentiated total.

**How to build it**
A feature and request-class tag on every logged call, joined to billing data at the same granularity.

**Moves**
Turns "the AI bill went up" into "feature X's retry rate went up," the only version of that sentence anyone can actually act on.

**Tradeoff**
Tagging discipline has to be enforced at every call site, or the untagged share becomes its own growing, unattributed bucket.

**Prove it**
Share of spend with a resolved feature tag, aiming for the high end.

#### Give each feature its own budget and alert threshold

In production

**What it changes**
Whether an emerging cost regression pages someone the hour it starts, or is discovered on next month's invoice.

**How to build it**
A per-feature spend ceiling with an alert threshold well under it, checked against actual spend on a short interval.

**Moves**
Converts a runaway feature into a same-day alert instead of a weeks-long gap between the regression starting and someone noticing it.

**Tradeoff**
Needs real per-feature spend data flowing in close to real time, which the tagging lever above is a prerequisite for.

**Prove it**
Time between a spend regression starting and an alert firing, measured against a real incident, not a drill.

#### Give every feature its own kill switch

Pre-launch

**What it changes**
Whether stopping one feature's spend requires a deploy, or a flag flip.

**How to build it**
A feature flag or rate limit that can throttle or disable one AI-backed feature without taking the rest of the product down with it.

**Moves**
Turns a runaway feature from an incident that runs until the next deploy window into one that stops within minutes.

**Tradeoff**
The flag has to be wired into the call path ahead of time. Retrofitting one mid-incident is already too late.

**Prove it**
Time to throttle a feature, tested as a drill rather than assumed to work.

#### Reconcile the model against the invoice, on a schedule

In production

**What it changes**
Whether the gap between the modeled estimate and the real bill is caught in a week, or in a quarter.

**How to build it**
A recurring reconciliation between the modeled per-outcome cost and actual billing data, owned by a named person, not left implicit.

**Moves**
This is the lever that catches every other lever's assumptions going stale: a cache that stopped hitting, a cascade whose escalation rate drifted, a source that quietly got re-added.

**Tradeoff**
Someone has to actually own this on a calendar, or it degrades back into a surprise on the invoice, the exact failure this page opened with.

**Prove it**
Modeled-vs-actual variance, tracked per reconciliation cycle.

[CostMon](https://customlabs.io/products/costmon/)[AI Cost Calculator](https://customlabs.io/tools/cost-calculator/)

No levers match that combination. Clear a filter to see more.

Five ways an estimate lies

## A cost model is a claim, not a fact.

Every one of these produces a confidently wrong number, not an obviously broken one.

### Sticker-price math

**Looks like:** A cost estimate multiplies a single blended $-per-token rate by a token count, as if input and output tokens cost the same.

**Costs you:** The estimate is wrong in a predictable direction, usually low, because output tokens are typically priced well above input tokens and a chatty feature spends most of its budget on the more expensive half.

**Fix:** Price the input and output halves of every request separately, using each one's own rate, before adding them back together.

[See the lever: Bound output length on purpose →](https://customlabs.io/cost/#bound-output-length)

### Happy-path volume

**Looks like:** A volume forecast counts real user requests only: no retries, no dev and staging traffic, no eval-suite runs against the same model.

**Costs you:** Every one of those omitted categories is a real, billed call. A retry-heavy feature can run several billed attempts per completed task even in a healthy system, none of which the happy-path forecast counted.

**Fix:** Model cost per completed task including every retry and fallback it actually took, not cost per request assuming the first attempt always lands.

[See the lever: Cap retry and fallback depth explicitly →](https://customlabs.io/cost/#cap-retry-and-fallback-depth)

### An assumed cache-hit rate

**Looks like:** The estimate applies a caching discount because the prompt "should" cache well, without ever pulling the actual cache-read number from provider usage data.

**Costs you:** A single volatile token near the front of the prompt is enough to void the discount entirely, and nothing in the system reports an error when that happens. The estimate keeps assuming a saving that stopped applying weeks ago.

**Fix:** Read the real cache-read share from provider usage data before building it into a forecast, and re-check it on a schedule, not once.

[See the lever: Measure the cache-read share, don't assume it →](https://customlabs.io/cost/#measure-actual-cache-read-share)

### Cost per request instead of cost per outcome

**Looks like:** The reported figure is spend divided by API calls made, when the business actually pays for, and cares about, resolved outcomes.

**Costs you:** A feature with a low failure rate and a feature that silently retries three times per success can report the identical cost-per-call number, while one of them is costing three times as much per outcome that actually lands.

**Fix:** Divide total spend by completed-task count, not call count, and track the ratio between the two over time.

[See the lever: Budget cost per completed task, not per call →](https://customlabs.io/cost/#budget-cost-per-outcome-not-per-call)

### A pilot curve extrapolated linearly

**Looks like:** A cost projection multiplies the pilot's per-user cost by the target user count, assuming cost scales in a straight line with volume.

**Costs you:** Context accumulation and agent fan-out both tend to grow faster than user count, not in step with it. A chat feature that resends growing history, or an agent workflow that spawns more sub-tasks under real load, both cost more per user at scale than the pilot ever showed.

**Fix:** Model the growth curve for context size and step count separately from the growth curve for user count, and extrapolate each on its own.

[See the lever: Give every agent run a hard ceiling →](https://customlabs.io/cost/#bound-every-agent-loop)

The scoreboard

## Six numbers, and how each one lies.

Every one of these is measurable today. None of them is trustworthy read alone. Pair it with the number next to it.

### Cost per resolved outcome

**Why it matters**
What the business actually pays for one real, completed task, not one API call.

**How it misleads**
On its own it says nothing about whether outcomes are actually landing; a feature can look cheap per outcome purely because it resolves so rarely that the denominator stays small.

**Pair with**
Resolution rate

### Cost per 1,000 requests

**Why it matters**
A volume-normalized number finance can forecast against committed traffic, independent of how big any one request happens to be.

**How it misleads**
A stable per-1,000-requests number can hide a request shape that got heavier: the same count of requests now carrying more tokens each.

**Pair with**
p95 tokens per request

### Cache-hit rate

**Why it matters**
Says whether the caching discipline this page argues for is actually landing rather than merely implemented.

**How it misleads**
A healthy-looking hit rate can still cover a small share of total spend, if the cached calls were cheap and the uncached ones are where the real money goes.

**Pair with**
Share of spend actually discounted

### Retry rate

**Why it matters**
The leading indicator for retry-amplified spend, visible before the invoice confirms it.

**How it misleads**
A low retry rate on a feature with a small number of very expensive retries can still mean most of the marginal spend is retries, even though the rate itself looks fine.

**Pair with**
Retry spend as a share of total spend

### Monthly run-rate

**Why it matters**
The number finance actually budgets against, updated as real traffic replaces the pilot estimate.

**How it misleads**
A flat run-rate over a few months can still be masking a volume increase offset by a routing improvement, two changes that look like no change at all until one of them stops.

**Pair with**
Forecast at committed volume

### Inference spend

**Why it matters**
The line item procurement and finance recognize first, and the one a per-token pricing page actually prices.

**How it misleads**
It excludes retrieval infrastructure, evals, observability, and the engineering time spent operating all of it. A system can look cheap on inference alone and expensive in total.

**Pair with**
Total cost of ownership, evals and infra and engineering included

What this is built from

## Verifiable, not claimed.

No invented benchmark numbers, no vendor price list. Just what's already documented on this site.

- The benchmark behind this page models cost per successful outcome across four common workload shapes, with every price, token count and multiplier stated so you can rederive each figure against your own traffic. [Insight: AI cost benchmark by workload](https://customlabs.io/insights/ai-cost-benchmark-by-workload/)
- The gap between a per-token sticker price and a production invoice was written up on this site before this page existed. That piece explains why the gap opens; the levers above are what closes it. [Insight: What an AI feature actually costs in production](https://customlabs.io/insights/what-ai-actually-costs/)
- Model cascade and stable-prefix prompt caching are patterns this site already documents in full. This page connects them, and the rest of the levers above, into one argument instead of leaving each as a standalone technique. [Pattern: Model cascade](https://customlabs.io/patterns/model-cascade/)
- Retry-amplified spend and prompt-cache-never-hits are named failure modes on this site, not hypotheticals. Each is the reason a specific lever above exists. [Failure mode: Retry-amplified spend](https://customlabs.io/failure-modes/retry-amplified-spend/)
- A published engagement cut blended cost per resolved request 41% with a model-cascade routing layer. That's one client's number, shown with its own baseline and window, not a rate to expect elsewhere. [Case study: Cutting inference spend without one vendor](https://customlabs.io/case-studies/model-agnostic-routing-cut-inference-spend/)
- The AI Cost Calculator models the multiplier this page argues for directly: retries, context growth, and eval overhead applied to your own volume, not a vendor sticker price. [AI Cost Calculator](https://customlabs.io/tools/cost-calculator/)
- CostMon exists because the studio hit its own version of the attribution-and-control surface above, running nine products' worth of AI, cloud, and SaaS spend across a dozen billing consoles with no shared total. [CostMon](https://customlabs.io/products/costmon/)
- FreeTier answers the surface-zero question before any of these levers apply: is the workload already free somewhere, before you build the paid alternative. [FreeTier](https://customlabs.io/products/freetier/)

### Sources

- [FinOps Foundation - FinOps Framework Overview](https://www.finops.org/framework/) The cost-management framework our cost model's phases map onto. Retrieved 2026-08-24.
- [FinOps Foundation - FinOps Capabilities](https://www.finops.org/framework/capabilities/) The capability list our cost levers are checked against. Retrieved 2026-08-24.

Not sure where your spend is actually going

A Ship Audit runs this model against your specific system and tells you which surface is actually costing you money, versus which one just looks incomplete on paper.

[Book a Ship Audit →](https://customlabs.io/diagnostic/ship-audit/) [See the Cost stage →](https://customlabs.io/handbook/cost/)

Questions

## Before you trust the estimate.

What teams ask us before they model AI spend.

01 Why is our inference bill so much higher than what we estimated? +

The estimate almost certainly priced one clean call per outcome, and production runs retries, fallbacks, and, for agents, several chained calls per completed task, none of which show up in a simple tokens-times-price model. See the five ways an estimate lies above; the gap is usually more than one of them at once.

02 What's the single fastest lever to pull first? +

Measuring cost per completed task instead of cost per call. It costs nothing to build beyond wiring a completion event through, and it's the lever that makes every other cost problem on this page visible instead of assumed.

03 Do we need all six surfaces on day one? +

No. Start with token demand and model routing, the two surfaces every feature has regardless of shape. Add caching once volume is real, retrieval-pipeline cost once there is a knowledge base to maintain, and the failure-economics and attribution surfaces once there is enough traffic for a regression to actually cost money.

04 Isn't cutting tokens just going to hurt quality? +

It can, which is why every lever on this page is checked against the eval suite, not shipped on the strength of the invoice alone. A cut that drops the golden-set pass rate isn't a cost win, it's a quality regression wearing a smaller bill.

05 How is this different from the AI Cost Calculator? +

The calculator turns your own volume and failure-rate assumptions into a number. This page is the levers that move that number once you have it: six surfaces, 24 named changes, and the five ways the estimate feeding the calculator can already be wrong before you touch a single lever.

06 Why no per-token price table? +

Per-token prices change every few weeks and a table of them would be the fastest-rotting page on this site within a quarter. This page talks in ratios, shapes, and levers that stay true regardless of what a provider charges this month, and points at the calculator for the arithmetic itself.

Put a number on it

You now know which six surfaces drive the bill, from token demand to attribution. Price your own workload against the same model, with every assumption shown.

[Open the cost calculator →](https://customlabs.io/tools/cost-calculator/) [See CostMon →](https://customlabs.io/products/costmon/)