The AI Cost Model
Where AI spend actually goes, the levers that move it, and the ways a cost estimate can already be wrong before you commit budget.
28 min read Read → Topics
AI Cost & Efficiency
Inference costInference cost is what it costs to run a trained model on a request. rarely matches the per-token sticker price once retries and context growth are counted. Fallback calls push it further. This is our work on modeling real AI spend and the build-vs-buy calls that keep it in check.
Start with The AI Cost Model if you own the AI budget. It's for engineers and finance leads who need to know where spend actually goes before it surprises them.
Guides#
Architectures#
Model-agnostic inference gateway
One call site in front of every model provider, so a price change or an outage is a config change, not a rewrite.
6 min read Read →Comparisons#
Open-Weight Models vs Frontier APIs: The Real Cost/Control Tradeoff
Self-hosting an open-weight model looks cheaper on paper than it behaves in practice. The real tradeoff against a frontier API, and when each one is right.
7 min read Read →Vector Database vs pgvector: Do You Actually Need a Dedicated Store
A dedicated vector database is a bigger commitment than most retrieval workloads need on day one. Here's how to tell if pgvector already covers your case.
8 min read Read →Insights#
What AI Actually Costs in Production, by Workload
A reproducible benchmark of cost per successful outcome across four common AI workloads, with every token assumption, price, and overhead multiplier shown.
9 min read Read →What an AI Feature Actually Costs in Production
Token costs that look trivial in a demo compound fast at scale. Here's how to make cost a first-class metric instead of a surprise on the invoice.
8 min read Read →Build, Buy, or Skip: A Framework for AI Decisions
A technically honest framework for deciding whether an AI initiative should be built in-house, bought off the shelf, or skipped entirely this year.
7 min read Read →Case studies#
Cutting Inference Spend 40% Without Betting on One Vendor
A Series B fintech's LLM spend outran revenue on one provider. Model-agnostic routing cut unit cost, vendor risk, and closed a data-residency gap.
7 min read Read →Patterns#
Model cascade
A cheap, fast model attempts every request first, and a validation signal decides whether that attempt is good enough to return. That signal can be a schema check, a self-reported confidence score, or a downstream eval. Only the fraction of requests the cheap model couldn't handle escalates to a stronger, pricier model. Most traffic never reaches the expensive tier at all.
Read →Stable-prefix prompt caching
The prompt is ordered with everything invariant across calls first: system instructions, tool definitions, few-shot examples. Everything that changes per request, such as retrieved chunks, user input, or timestamps, goes last. Prompt caching only pays off when the shared prefix is byte-identical across requests. One volatile token near the front is enough to bust the cache for the entire call, so ordering invariant-first is what makes the discount actually apply.
Read →Failure modes#
Retry-amplified spend
The estimate priced the happy path: one clean call per outcome. Production reality includes retries on malformed or rate-limited calls, fallbacks to a larger model when the first attempt fails, and agent loops that make several calls per completed task. Every one of those multiplies calls per successful outcome without multiplying the original per-token estimate.
Read →Prompt cache never hits
A dynamic prefix (a timestamp, a per-user greeting, a reordered tool list, retrieved chunks placed before the static instructions) changes the start of the prompt on every call. Prompt caching only pays off when the shared prefix is byte-identical across requests, and one volatile token near the front is enough to bust the whole cache.
Read →Tools#
AI Cost Calculator
What AI actually costs to run in production — not the sticker price.
Read →