Cutting Inference Spend 40% Without Betting on One Vendor

The problem

A Series B fintech running fraud-risk scoring and case-note summarization on a single LLM provider had a spend curve outpacing its revenue curve. Every new risk model, every new customer segment, every new use case added to the same provider bill, and the finance team had started asking the question that eventually stops every all-in vendor bet: what happens if this provider raises prices, degrades service, or has an outage during a fraud spike?

The concentration risk wasn’t hypothetical. The provider had already pushed a price increase mid-contract, and a multi-hour outage six months earlier had forced the fraud team to fall back to a slower, rules-only model for half a business day, during which a measurable share of legitimately fraudulent transactions cleared. On top of that, the company had signed a European enterprise customer whose contract required certain risk-scoring workloads to run on infrastructure physically located in the EU. The incumbent provider had no committed in-region deployment on the fintech’s timeline. The team was boxed in: pay whatever the single vendor asked, absorb outages as a cost of doing business, and turn down or delay a signed deal.

What we ruled out, and what we built

Two paths looked easy and were both wrong for a Series B company. The first was staying single-vendor and negotiating harder on price. That doesn’t touch the concentration risk, and a startup has little real leverage against a provider it depends on for a mission-critical workload. The second was self-hosting open-weight models to escape provider pricing entirely. That’s technically sound, but it trades a vendor dependency for a GPU-fleet and MLOps burden that a company this size shouldn’t be carrying as a differentiator. Fraud-risk scoring isn’t where this team wanted to spend its infrastructure headcount.

We designed a model-agnostic routing layer instead: one internal interface that every product surface calls, with the routing logic — not the calling code — deciding which model handles a given request. Requests are bucketed into classes (fraud scoring, case-note summarization, customer-facing explanation text, and so on), and each class has its own evaluation suite built from labeled historical cases with known-correct outcomes. For each class, the router sends traffic to the cheapest model that clears that class’s eval bar, not the cheapest model overall, because a model that’s cheap and wrong on fraud scoring is not actually cheap. Underneath the router sits a provider fallback chain, so a degraded or unavailable primary model fails over automatically to a secondary that has already passed the same eval gate, instead of falling back to a slower rules engine as a manual last resort.

The EU residency requirement became a proof point for the architecture rather than a separate project. Because every call already went through the router’s interface rather than a hardcoded SDK call to one provider, standing up an in-region model for the residency-bound workload class was a routing-config change: point that request class at the new provider, run it against the existing eval suite to confirm it cleared the bar, and cut over. No product code touched a provider SDK directly, so there was nothing in the application layer to rewrite.

We also built cost and quality into the same feedback loop instead of treating them as separate concerns. The router logs which model handled each request, its latency, its cost, and its eval-suite pass/fail status, feeding a dashboard the engineering team checks weekly. When a provider drops a cheaper model that clears the eval bar for a given class, the router picks it up automatically and cost drops without anyone touching the fraud-scoring logic itself.

The outcome

Blended cost per resolved request fell 41% in the first full month after cutover, compared to the trailing 30-day baseline on the single-vendor setup. The router settled on cheaper, eval-passing models for the request classes that didn’t need the most expensive option. p95 latency dropped 28% over the same window, largely because several request classes moved to smaller, faster models that still cleared their eval bar; the most expensive model was rarely the fastest one either. The EU residency migration took three weeks end to end, with zero downtime and zero changes to product code. The router config change was the entire migration.

For the CFO, the story is a cost curve that finally decouples from usage growth, with a payback period inside a single quarter once the routing layer’s build cost is weighed against the ongoing savings. A sub-quarter payback is the kind of number that gets infrastructure work approved without a fight. For IT and security, the router gave them one audited egress point and one place to manage provider credentials instead of scattered SDK calls, and it closed the specific compliance gap that had been blocking the EU deal, without opening a new one: no lock-in, and no single point of failure holding a regulatory requirement hostage. For the operations team running fraud detection day to day, the fallback chain converts what used to be a single-provider outage — the kind that forced a costly manual fallback during the earlier incident — into a routine failover event that doesn’t reach a pager. Just as important, the eval gate means a cheaper model can never silently ship a quality regression into fraud scoring; nothing gets routed to a request class unless it has already proven it clears that class’s bar, so cost optimization and score quality move together instead of trading off against each other.

The team now treats every new model release from every provider as an additional candidate to test against its eval suites, not a migration decision. If it clears the bar and it’s cheaper or faster, the router starts using it. If it doesn’t, nothing changes. That’s the leverage the VP of Engineering was pointing to: the fintech no longer negotiates with one provider from a position of dependency, because switching, in the architecture that actually matters, is a config change.