Model-agnostic inference gateway
One call site in front of every model provider, so a price change or an outage is a config change, not a rewrite.
“We're calling one provider's SDK directly from a dozen places, and we want to be able to switch or split traffic without touching all of them.”
What does the system look like?#
How does a request move through it?#
- Call one interface, not one provider
Application code issues a model-agnostic request; which provider actually serves it is the gateway's decision, not the caller's.
- Route by a cascade policy
A cheap, fast model attempts every request first; a validation or confidence signal decides whether that is good enough or the request escalates to a stronger, pricier rung.
- Check the prompt cache before spending a full call
Static instructions and tool definitions sit first in the prompt, per-request content last, so a byte-identical prefix actually gets the caching discount.
- Normalize through a provider adapter
The adapter translates the gateway's internal request shape into whichever SDK the chosen provider actually speaks, and normalizes the response back.
- Fall back on failure, not silence
A provider outage or an error climbs a defined fallback ladder to the next provider, logged as a fallback event, not swallowed as a retry nobody sees.
- Report cost per call, to one ledger
Every call, on every provider, reports its actual cost back to one place, so cost per completed task is a query, not a monthly reconciliation project.
What are the pieces, and what breaks without each one?#
Provider adapter
Normalize the gateway's internal request and response shape against whichever SDK a given provider actually speaks.
Breaks without it: Every provider swap becomes a rewrite of application code instead of a config change in one place.
Routing & cascade policy
Attempt cheap first, validate the result, and escalate to a stronger model only on a real failure or low-confidence signal.
Breaks without it: Every request pays frontier prices even for the easy majority a cheap model would have handled correctly.
Prompt cache
Order the prompt invariant-first so a byte-identical prefix actually gets the provider's caching discount.
Breaks without it: A single volatile token near the front of the prompt, a timestamp, a request ID, silently busts the cache on every call with no visible symptom.
Fallback ladder
Define what happens on a provider outage or error: a named secondary provider, a degraded response, or an explicit unavailable state.
Breaks without it: A provider's next incident becomes a live outage discovery instead of a config path that was already decided in advance.
Cost telemetry
Log the real cost of every call, on every provider, into one ledger keyed by feature and outcome.
Breaks without it: The actual invoice arrives as a surprise three times the modeled estimate, discovered a month after the spend already happened.
Where does this need a decision, not a default?#
| Decision | Default choice | Why |
|---|---|---|
| The gateway itself | A thin abstraction you build and own around the two or three providers you actually use | You do not need a general-purpose router product to get the model-agnostic benefit; you need the one abstraction between your application and whichever provider currently serves a given call. |
| Open-weight rung | A hosted open-weight endpoint before self-hosted GPU infrastructure, until volume justifies operating your own | Self-hosting inference is a real ops commitment, worth taking on once volume and cost make it pay off, not before. |
| Cost visibility | A dedicated cost tool over a hand-rolled spreadsheet reconciliation, once you're running more than one or two providers | Reconciling a dozen billing formats by hand is exactly the kind of manual process a tool should own instead. CostMon is our own answer to this, built because we hit the same reconciliation problem running our own stack; it's one option, not the only one that works. |
| Caching implementation | Ordering discipline in your own prompt templates first, before reaching for a managed caching proxy | The most common cache-miss cause is a template that puts something volatile near the front; fixing the template usually gets most of the discount before any extra infrastructure is needed. |
What actually drives the bill?#
- Frontier-tier tokens for the hard tail the cascade escalates to
- Cache-miss rate on the prompt prefix that is supposed to be stable
- Fallback calls stacking on top of the original failed attempt
Dominant cost Whichever provider sits at the top of the cascade, because that tier is priced at the premium rate and the escalation rate directly sets how much traffic pays it.
The lever A well-calibrated model cascade to keep most traffic on the cheap rung, paired with stable-prefix caching so the discount that is already available actually gets claimed.
Model your own numbers with the AI Cost Calculator →How do you know it actually works?#
Contract check across every provider
Confirms the normalized response shape holds regardless of which provider actually served the call.
Cost per completed task, tracked per provider and rung
Separates the modeled estimate from what a cascade and fallback ladder actually cost in production.
Drift detection on a live-traffic sample
Catches a provider-side model version change that silently shifts quality on one rung of the cascade.
What will your reviewers ask about this?#
Documented exit path per provider
A named alternative provider the system can route to without a rewrite, plus a data-export procedure for anything stored with the current one.
Fallback behavior defined ahead of an incident
What happens on an outage or breaking change is decided in the routing layer before the vendor's next incident, not during it.
Recurring vendor audit-rights check
Each provider's current compliance evidence is re-pulled on a schedule, not checked once at signing and forgotten.
Which patterns and failure modes tie in?#
How long does a first version actually take?#
2-4 weeks for a thin gateway sitting in front of one existing call site; 6-10 weeks once a real cascade, a second provider, and cost telemetry are all live.
When is this the wrong shape?#
- You call one model, from one provider, for one feature, with no plan to add a second. The abstraction is overhead with nothing yet to route between.
- Your call volume is too low for the caching or cascade savings to matter. The gateway adds a hop for a saving that rounds to zero at that scale.
- You need a fully managed answer today with no infrastructure of your own to operate. Reconsider buy over build in that case, rather than building a gateway you don't actually want to run.
Sources
- FinOps Foundation - FinOps Framework Overview
The cost-management framework our cost model's phases map onto. Retrieved 2026-08-24.
- FinOps Foundation - FinOps Capabilities
The capability list our cost levers are checked against. Retrieved 2026-08-24.
A Ship Audit checks which of these an existing or planned system actually needs, against what is most likely to break first.