The AI Cost Model
Where AI spend goes, the levers that move it, and the ways a cost estimate can be wrong before you commit budget.
27 min read Read → Topics
AI Cost & Efficiency
Inference costInference cost is what it costs to run a trained model on a request. rarely matches the per-token sticker price once retries and context growth are counted. Fallback calls push it further. This is our work on modeling real AI spend and the build-vs-buy calls that keep it in check.
Start with The AI Cost Model if you own the AI budget. It's for engineers and finance leads who need to know where spend actually goes before it surprises them.
Guides#
Architectures#
Model-agnostic inference gateway
One call site in front of every model provider. A price change or an outage becomes a config change.
6 min read Read →Comparisons#
Open-Weight Models vs Frontier APIs: The Real Cost/Control Tradeoff
Self-hosting an open-weight model looks cheaper on paper than it behaves in practice. The real tradeoff against a frontier API, and when each one is right.
7 min read Read →Vector Database vs pgvector: Do You Need a Dedicated Store
A dedicated vector database is a bigger commitment than most retrieval workloads need on day one. This piece shows how to tell if pgvector covers your case.
8 min read Read →Insights#
AI Cost per Outcome in Production, by Workload
A reproducible benchmark of cost per successful outcome across four common AI workloads, with every token assumption, price, and overhead multiplier shown.
9 min read Read →What an AI Feature Costs in Production
Token costs that look trivial in a demo compound fast at scale. Make cost a first-class metric instead of a surprise on the invoice.
7 min read Read →Build, Buy, or Skip: A Framework for AI Decisions
A technically honest framework for deciding whether an AI initiative should be built in-house, bought off the shelf, or skipped entirely this year.
7 min read Read →Case studies#
Cutting Inference Spend 40% Without Betting on One Vendor
A Series B fintech's LLM spend outran revenue on one provider. Model-agnostic routing cut unit cost, vendor risk, and closed a data-residency gap.
7 min read Read →Patterns#
Model cascade
A cheap, fast model tries every request first. A validation signal decides whether that answer is good enough to return. Only failed requests escalate to a stronger, pricier model. The validation signal can be a schema check or a self-reported confidence score. A downstream eval also works. Most traffic never reaches the expensive tier.
Read →Stable-prefix prompt caching
Put invariant content first in the prompt and per-request content last, so the shared prefix stays byte-identical and the cache hits. Prompt caching pays off only when the shared prefix is byte-identical across requests. One volatile token near the front busts the cache for the entire call. System instructions and tool definitions belong in the invariant part. Few-shot examples belong there too. Retrieved chunks and user input go last. A timestamp goes there as well.
Read →Failure modes#
Retry-amplified spend
The estimate priced the happy path: one clean call per outcome. Production reality includes retries on malformed or rate-limited calls. It includes fallbacks to a larger model when the first attempt fails. It also includes agent loops that make several calls per completed task. Every one of those multiplies calls per successful outcome without multiplying the original per-token estimate.
Read →Prompt cache never hits
A dynamic prefix changes the start of the prompt on every call. Examples are a timestamp, a per-user greeting, a reordered tool list, retrieved chunks placed before the static instructions. Prompt caching only pays off when the shared prefix is byte-identical across requests. One volatile token near the front busts the whole cache.
Read →Tools#
AI Cost Calculator
What AI costs to run in production.
Read →