How do you avoid paying frontier-model prices for every request?
A cheap, fast model attempts every request first. A validation or confidence signal — a schema check, a self-reported confidence score, a downstream eval — decides whether that attempt is good enough to return, and only escalates to a stronger, pricier model on the fraction of requests the cheap one couldn't handle. Most traffic never reaches the expensive tier at all.
Also known as cheap-first routing, model escalation ladder, confidence-based upgrade
What problem does this solve?
A single fixed model for every request prices for the hardest case in the distribution, even though most real requests are easier than that. Defaulting to a frontier model everywhere is the simplest thing to build, but it means paying frontier rates on the classification task that a much cheaper model would have gotten right anyway.
The naive alternative — picking one "good enough" cheaper model for everything — trades the opposite way: it under-serves the harder tail of requests that genuinely need the stronger model's capability, and that failure is often invisible until someone notices quality is inconsistent without an obvious pattern.
How does it work?
Every request first goes to a cheap, fast model. A validation step checks whether that attempt is actually good enough: a schema or structural check the output must pass, a confidence score the model reports about its own answer, or a lightweight downstream check specific to the task, such as whether an extracted value looks valid or a classification meets a confidence floor.
Requests that pass validation return immediately at the cheap tier's cost. Requests that fail, or fall under the confidence floor, escalate to the next, stronger and pricier model in the ladder, which repeats the same attempt with more capability behind it. The ladder can have more than two rungs, escalating further only as needed.
This only works because the validation signal is real and calibrated — a confidence score that's poorly calibrated, near-uniformly high as with a lenient judge, either escalates almost everything, erasing the savings, or almost nothing, silently under-serving the hard tail. The escalation rate itself becomes the metric that tells you whether the ladder is tuned correctly.
What are the moving parts?
- Route every request to the cheapest rung first
Attempt every request at the lowest-cost model in the ladder before considering escalation.
- Validate the cheap attempt
Check it against a schema, a confidence score, or a task-specific correctness check.
- Return immediately on a pass
A validated cheap-tier result ships at the cheap tier's cost, with no further calls.
- Escalate on a fail or low confidence
Route to the next stronger, pricier model in the ladder when validation fails.
- Repeat validation at each rung
Escalate further only as needed, rather than jumping straight to the top of the ladder.
- Track escalation rate per rung
Use it as the primary signal for whether the cascade is tuned correctly.
When does it fit, and when doesn't it?
Use it when
- Request difficulty genuinely varies — a meaningful share of traffic is easy enough for a cheap model to handle correctly
- There is a real, checkable validation signal available (schema conformance, a confidence score, a structural check) to decide pass/escalate
- Inference cost is a measured, material line item, not a rounding error against total spend
- The task tolerates a small latency increase on the escalated fraction, since the cheap attempt runs first, adding a step before escalation
Don't use it when
- Nearly every request genuinely needs frontier-model capability — the cascade adds latency and complexity with almost nothing escalating to save
- You have no reliable validation signal to gate escalation on — an uncalibrated gate either escalates everything (no savings) or nothing (silently wrong answers at the cheap tier)
- Latency is the binding constraint and even the cheap-tier attempt's added round trip before a possible escalation isn't affordable
What does it actually cost?
Most traffic resolves at the cheap tier's cost, often a large fraction of the frontier price
Requests that do escalate pay the cheap attempt's cost on top of the expensive one — escalation isn't free, it's additive
Quality on the easy majority stays high while spend drops
Building and calibrating a real validation/confidence gate is its own project — an uncalibrated one defeats the pattern entirely
The ladder can extend to more than two rungs as needed, tuning cost/quality finely
More rungs means more escalation logic and more places for a miscalibrated gate to misroute a request
Escalation rate becomes a direct, tunable dial on the cost/quality tradeoff
Adds a small latency tax to every escalated request, since the cheap attempt still has to run and fail first
How do you know it's working?
- Escalation rate per rung, tracked as the primary health metric for the whole cascade
- Cost per completed task, blended across all rungs, vs. a flat single-model baseline
- Quality/accuracy at each rung, measured independently, not just at the final returned answer
- Added latency from the cheap-tier attempt on requests that ultimately escalate
What failure modes does this prevent?
A Ship Audit checks which of these patterns your system actually needs, prioritized against what's most likely to break first.