Model-agnostic inference gateway
One call site in front of every model provider. A price change or an outage becomes a config change.
“We call one provider's SDK directly from a dozen places. We want to switch or split traffic without touching all of them.”
What does the system look like?#
How does a request move through it?#
- Call one interface
Application code issues a model-agnostic request. The gateway decides which provider serves it. The caller does not decide.
- Route by a cascade policy
A cheap, fast model attempts every request first. A validation or confidence signal decides if the result is good enough. If not, the request escalates to a stronger, pricier rung.
- Check the prompt cache before spending a full call
Static instructions and tool definitions sit first in the prompt. Per-request content sits last. A byte-identical prefix then gets the caching discount.
- Normalize through a provider adapter
The adapter translates the gateway's internal request shape into the chosen provider's SDK. It then normalizes the response back.
- Fall back on failure
A provider outage or error climbs a defined fallback ladder to the next provider. The gateway logs it as a fallback event. No retry stays hidden.
- Report cost per call, to one ledger
Every call on every provider reports its cost back to one place. Cost per completed task becomes a query. It no longer needs a monthly reconciliation project.
What are the pieces, and what breaks without each one?#
Provider adapter
Normalize the gateway's internal request and response shape against each provider's SDK.
Breaks without it: Every provider swap forces a rewrite of application code. With an adapter, it is a config change in one place.
Routing & cascade policy
Attempt the cheap model first and validate the result. Escalate to a stronger model only on a failure or a low-confidence signal.
Breaks without it: Every request pays frontier prices. That includes the easy majority that a cheap model would handle correctly.
Prompt cache
Order the prompt with invariant content first. A byte-identical prefix then gets the provider's caching discount.
Breaks without it: One volatile token near the front of the prompt busts the cache on every call. A timestamp or a request ID is enough. No visible symptom appears.
Fallback ladder
Define what happens on a provider outage or error. Choose a named secondary provider, a degraded response, or an explicit unavailable state.
Breaks without it: The next incident at a provider becomes a live outage discovery. With a ladder, the config path was decided in advance.
Cost telemetry
Log the real cost of every call on every provider into one ledger. Key the ledger by feature and outcome.
Breaks without it: The invoice arrives at three times the modeled estimate. The team finds out a month after the spend.
Which choices need a decision?#
| Decision | Default choice | Why |
|---|---|---|
| The gateway itself | A thin abstraction that you build and own around the two or three providers you use | You do not need a general-purpose router product to get the model-agnostic benefit. You need one abstraction between your application and the provider that serves each call. |
| Open-weight rung | A hosted open-weight endpoint first. Move to self-hosted GPU infrastructure when volume justifies operating your own | Self-hosting inference is a real ops commitment. Take it on once volume and cost make it pay off. |
| Cost visibility | A dedicated cost tool in place of a hand-rolled spreadsheet reconciliation, once you run more than one or two providers | A tool should own the reconciliation of a dozen billing formats. CostMon is our own answer to this. We built it after we hit the same problem on our own stack. It is one option among several. |
| Caching implementation | Ordering discipline in your own prompt templates first. Reach for a managed caching proxy later | The most common cause of a cache miss is a template with volatile content near the front. Fixing the template usually wins most of the discount. No extra infrastructure is needed. |
What drives the bill?#
- Frontier-tier tokens for the hard tail that the cascade escalates
- Cache-miss rate on the prompt prefix that should be stable
- Fallback calls stacking on top of the original failed attempt
Dominant cost The provider at the top of the cascade. That tier has the premium rate. The escalation rate sets how much traffic pays it.
The lever A well-calibrated model cascade keeps most traffic on the cheap rung. Stable-prefix caching claims the discount that is already available.
Model your own numbers with the AI Cost Calculator →How do you know it works?#
Contract check across every provider
Confirms that the normalized response shape holds, whichever provider served the call.
Cost per completed task, tracked per provider and rung
Separates the modeled estimate from the production cost of the cascade and the fallback ladder.
Drift detection on a live-traffic sample
Catches a provider-side model version change that shifts quality on one rung of the cascade.
What will your reviewers ask about this?#
Documented exit path per provider
A named alternative provider that the system can route to without a rewrite. A data-export procedure covers anything stored with the current provider.
Fallback behavior defined ahead of an incident
The routing layer defines what happens on an outage or breaking change. The team decides this before the vendor's next incident.
Recurring vendor audit-rights check
The team re-pulls each provider's current compliance evidence on a schedule. A single check at signing is not enough.
Which patterns and failure modes tie in?#
How long does a first version take?#
2-4 weeks for a thin gateway in front of one existing call site. 6-10 weeks once a real cascade, a second provider, and cost telemetry are all live.
When is this the wrong shape?#
- You call one model from one provider for one feature. You have no plan to add a second. The abstraction is overhead with nothing to route between.
- Your call volume is too low for caching or cascade savings to matter. The gateway adds a hop. At that scale the saving rounds to zero.
- You need a fully managed answer today with no infrastructure of your own to operate. Consider buying in that case. Do not build a gateway that you do not want to run.
Sources
- FinOps Foundation - FinOps Framework Overview
The cost-management framework our cost model's phases map onto. Retrieved 2026-08-24.
- FinOps Foundation - FinOps Capabilities
The capability list our cost levers are checked against. Retrieved 2026-08-24.
A Ship Audit checks which of these an existing or planned system needs. It ranks them by what is most likely to break first.