Cutting Inference Spend 40% Without Betting on One Vendor

The problem

A Series B fintech running fraud-risk scoring and case-note summarization on a single LLM provider had a spend curve outpacing its revenue curve. Every new risk model added cost. So did every new customer segment and every new use case. All of it landed on the same provider’s bill. The finance team asked the question that stops any all-in vendor bet. What happens if this provider raises prices? What if it degrades service, or goes down during a fraud spike?

The concentration risk was not hypothetical. The provider had already pushed a price increase mid-contract. A multi-hour outage six months earlier forced the fraud team to fall back to a slower, rules-only model for half a business day. During that time, a measurable share of legitimately fraudulent transactions cleared. The company had also signed a European enterprise customer. Its contract required certain risk-scoring workloads to run on infrastructure physically located in the EU. The incumbent provider had no committed in-region deployment on the fintech’s timeline. The team was boxed in. It could pay whatever the vendor asked. It could absorb outages as a cost of doing business. Or it could turn down or delay a signed deal.

What we ruled out, and what we built

Two paths looked easy and were both wrong for a Series B company. The first was staying single-vendor and negotiating harder on price. That does not touch the concentration risk, and a startup has little real leverage against a provider it depends on for a mission-critical workload. The second was self-hosting open-weight models to escape provider pricing entirely. That is technically sound. But it swaps a vendor dependency for a GPU fleet and an MLOps burden this company should not carry as a differentiator. Fraud-risk scoring was not where this team wanted to spend its infrastructure headcount.

We designed a model-agnostic routing layer instead. Every product surface calls one internal interface. The routing logic, not the calling code, decides which model handles each request. Requests fall into classes: fraud scoring, case-note summarization, customer-facing explanation text, and more. Each class has its own evaluation suite, built from labeled historical cases with known-correct outcomes. For each class, the router sends traffic to the cheapest model that clears that class’s eval bar. It does not send traffic to the cheapest model overall, because a model that is cheap and wrong on fraud scoring is not cheap. Underneath the router sits a provider fallback chain. A degraded or unavailable primary model fails over automatically to a secondary model. That secondary has already passed the same eval gate. So the system does not fall back to a slower rules engine as a manual last resort.

The EU residency requirement became a proof point for the architecture, not a separate project. Every call already went through the router’s interface, not a hardcoded SDK call to one provider. So standing up an in-region model for the residency-bound workload class took one routing-config change. The team pointed that request class at the new provider. They ran it against the existing eval suiteAn eval suite is a versioned set of test cases that measures whether outputs are good. to confirm it cleared the bar. Then they cut over. No product code touched a provider SDK directly, so nothing in the application layer needed a rewrite.

We also built cost and quality into the same feedback loop, instead of treating them as separate concerns. The router logs which model handled each request. It also logs the latency, the cost, and the eval-suite pass/fail status for that request. That data feeds a dashboard the engineering team checks weekly. When a provider ships a cheaper model that clears the eval bar for a class, the router picks it up automatically. Cost drops without anyone touching the fraud-scoring logic itself.

The outcome

Blended cost per resolved request fell 41% in the first full month after cutover, compared to the trailing 30-day baseline on the single-vendor setup. The router settled on cheaper, eval-passing models for the request classes that did not need the most expensive option. p95 latency dropped 28% over the same window. Several request classes moved to smaller, faster models that still cleared their eval bar. The most expensive model was rarely the fastest one either. The EU residency migration took three weeks end to end, with zero downtime and zero changes to product code. The router config change was the entire migration.

For the CFO, the story is a cost curve that finally decouples from usage growth. The routing layer’s build cost pays back inside a single quarter against the ongoing savings. A sub-quarter payback is the kind of number that gets infrastructure work approved without a fight. For IT and security, the router gave them one audited egress point. It also gave them one place to manage provider credentials, instead of scattered SDK calls. It closed the compliance gap that had been blocking the EU deal, without opening a new one. There is no lock-in, and no single point of failure holding a regulatory requirement hostage. For the operations team running fraud detection, the fallback chain changes what used to be a single-provider outage into a routine failover event. That is the same kind of outage that forced a costly manual fallback during the earlier incident. Now it does not reach a pager. Just as important, the eval gate means a cheaper model can never silently ship a quality regression into fraud scoring. Nothing routes to a request class until it has already proven it clears that class’s bar. Cost and score quality move together instead of trading off against each other.

The team now treats every new model release from every provider as another candidate. Each one gets tested against the eval suites, not treated as a migration decision. If it clears the bar and it is cheaper or faster, the router starts using it. If it does not, nothing changes. That is the leverage the VP of Engineering meant. The fintech no longer negotiates with one provider from a position of dependency. Switching, in the architecture that matters, is a config change.