How do you avoid paying frontier-model prices for every request?
A cheap, fast model tries every request first. A validation signal decides whether that answer is good enough to return. Only failed requests escalate to a stronger, pricier model. The validation signal can be a schema check or a self-reported confidence score. A downstream eval also works. Most traffic never reaches the expensive tier.
Also known as cheap-first routing, model escalation ladder, confidence-based upgrade
What problem does this solve?#
A single fixed model prices every request for the hardest case. Most real requests are easier than that. Defaulting to a frontier model everywhere is the simplest design to build. It also means paying frontier rates for a classification task that a much cheaper model would answer correctly.
The alternative is one cheaper model that seems good enough for everything. That choice fails the other way. It under-serves the harder tail of requests that need the stronger model. Nobody notices until quality looks inconsistent with no obvious pattern.
How does it work?#
Every request goes first to a cheap, fast model. A validation step then checks that attempt. The check can be a schema or structural test on the output. It can be a confidence score that the model reports about its own answer. A lightweight check specific to the task also works, such as a valid extracted value or a classification above a confidence floor.
Requests that pass validation return at the cheap tier's cost. Requests that fail, or fall under the confidence floor, escalate to the next model in the ladder. That model is stronger and pricier. It repeats the same attempt with more capability. The ladder can have more than two rungs. Each request escalates only as far as it needs.
This works only when the validation signal is calibrated. A lenient judge gives near-uniformly high confidence scores. A poorly calibrated gate then escalates almost everything and erases the savings. Or it escalates almost nothing and under-serves the hard tail without any warning. The escalation rate tells the team whether the ladder is tuned correctly.
What are the moving parts?#
- Route every request to the cheapest rung first
Attempt each request at the lowest-cost model in the ladder before any escalation.
- Validate the cheap attempt
Check the answer against a schema or a confidence score. A task-specific correctness check also works.
- Return immediately on a pass
A validated cheap-tier result ships at the cheap tier's cost. It needs no further calls.
- Escalate on a fail or low confidence
Route to the next stronger, pricier model in the ladder when validation fails.
- Repeat validation at each rung
Climb one rung at a time until a result passes.
- Track escalation rate per rung
Use this rate as the primary signal that the cascade is tuned correctly.
When does it fit, and when doesn't it?#
Use it when
- Request difficulty varies, and a meaningful share of traffic is easy enough for a cheap model to answer correctly
- A checkable validation signal exists to decide pass or escalate, such as schema conformance or a confidence score
- Inference costInference cost is what it costs to run a trained model on a request. is a measured, material line item in total spend
- The task tolerates a small latency increase on the escalated fraction, because the cheap attempt adds a step before escalation
Don't use it when
- Nearly every request needs frontier-model capability. The cascade then adds latency and complexity, and almost nothing escalates to save money
- No reliable validation signal exists to gate escalation. An uncalibrated gate escalates everything and saves nothing. Or it escalates nothing and returns wrong answers from the cheap tier
- Latency is the binding constraint, and the extra round trip of a cheap-tier attempt is not affordable
What does it cost?#
Most traffic resolves at the cheap tier's cost, often well below the frontier price
Escalated requests pay for the cheap attempt plus the expensive one. The two costs add up
Quality on the easy majority stays high while spend drops
Building and calibrating a validation gate is its own project. An uncalibrated gate defeats the pattern
The ladder can grow past two rungs, which tunes cost and quality finely
More rungs need more escalation logic. They also add places where a miscalibrated gate can misroute a request
The escalation rate gives the team a direct dial on the cost and quality tradeoff
Every escalated request pays a small latency tax, because the cheap attempt must run and fail first
How do you know it's working?#
- Escalation rate per rung, tracked as the primary health metric of the cascade
- Blended cost per completed task across all rungs, compared with a flat single-model baseline
- Quality and accuracy at each rung, measured independent of the final returned answer
- Added latency from the cheap-tier attempt on requests that end up escalating
What failure modes does this prevent?#
Sources
- FinOps Foundation - FinOps Framework Overview
The cost-management framework our cost model's phases map onto. Retrieved 2026-08-24.
- FinOps Foundation - FinOps Capabilities
The capability list our cost levers are checked against. Retrieved 2026-08-24.
A Ship Audit checks which of these patterns your system needs. It ranks them by what is most likely to break first.