What an AI Feature Costs in Production

An AI feature’s real cost isn’t the per-token sticker price. The real number is cost per successful user outcome at real traffic, after every retry, every fallback call, and every prompt that grew. That number typically runs five to twenty times the sticker price. Cost behaves like a runtime property. It drifts as usage and models change. Instrument it on every call. Assert on it in the eval suite. Review the curve on a schedule.

The gap isn’t dishonesty or bad math. A demo bills a handful of clean calls against one API key. Nobody models what happens when that call runs thousands of times a day against real, messy input. Cost deserves the same ongoing attention as latency or error rate. Checking it once at launch and moving on isn’t enough.

Why is the per-token price the wrong number?

Token counts compound in ways a demo hides. A single test call with a short prompt and a short answer looks cheap in isolation. Production input runs longer: real documents, real conversation history, real edge cases that need more context to resolve. Many features re-send that growing context on every turn. A chat feature that starts a session at 500 input tokens can be paying for 5,000 tokens of repeated context by turn ten. That growth stays invisible until someone graphs cost per session instead of cost per call.

Latency and cost pull in opposite directions. The fix for a slow response is often a smaller, faster model or a shorter prompt. The fix for a low-quality response is often a bigger model or more context. Every AI feature sits somewhere on that tradeoff. Treat it as fixed instead of tunable, and you stay stuck with the choice you made in week one, even after requirements change. Teams that manage this well track quality, latency, and cost as three dials on one feature, each owned by the same reviewer.

Retries and fallbacks bill twice, too. A rate limit, a malformed tool call, a model that returns something the parser can’t handle: each of these triggers a retry. A retry means paying for the same request twice while the user waits for the second attempt. If your fallback chain routes to a second provider on failure, you now pay two different pricing schedules for one user action. None of this shows up in a per-call price estimate. All of it shows up in the actual bill.

Nobody notices a slow price change. Providers adjust pricing, deprecate cheaper model tiers, and change what counts as an input versus output token. This happens regularly, often several times a year. A cost model built once at launch and never revisited will be wrong within a quarter. Nobody made a mistake; the ground it was built on moved.

We put concrete, modeled numbers on this gap in AI cost per outcome, by workload. It is a reproducible cost-per-successful-outcome benchmark across four common workloads, with every price, token count, and multiplier shown.

How do you make cost a first-class metric?

The fix looks the same as the fix for any other production concern. Instrument it, alert on it, and put it in the same review as correctness. Concretely, log token count, model, and dollar cost on every model call, tagged by feature and by user. Then “what does this cost” becomes a query instead of a guess. Put a cost assertion in the eval suiteAn eval suite is a versioned set of test cases that measures whether outputs are good. alongside the accuracy assertion. A prompt change that triples token usage to improve quality then shows up as a tradeoff decision, not a silent win. Review the cost curve against usage growth on a schedule, not only when finance asks a pointed question.

Before you write a line of code, run the back-of-envelope version yourself. Model what it will cost at your expected traffic, and see where the multiplier comes from.

We built CostMon because this same problem shows up one level up, across an entire stack rather than a single feature. Spend on AI, cloud, and SaaS scatters across a dozen billing consoles. Each has its own units and its own delay before a charge shows up. So “what are we spending” ends up as a guess assembled from memory, not a number anyone can point to. The instinct behind it applies inside a single AI feature too: cost is knowable. It’s just rarely tracked with the rigor teams give the metrics they watch.

How does this fit into a client engagement?

When we build an AI feature, cost instrumentation goes in with the feature itself. It is never a follow-up ticket after the first surprising invoice. That means per-call cost logging from day one, and a cost ceiling agreed with the client before launch rather than discovered afterward. It also means cost sits alongside quality and latency in the numbers we report at handover, tracking the same five-to-twenty-times multiplier. It’s a small amount of extra work up front. One team can answer “is this still worth running at this volume” with data. The other team finds the answer only when someone reads the invoice line by line.

Two mechanisms drive a bill that runs ahead of the estimate: see the field guide entries on retry-amplified spend and prompt cache never hits.

The full version of this argument lives in the AI cost model: six surfaces where spend hides, and the named levers that move each one.