Retrieval Pipeline That Actually Cut Support Load

The problem

Their in-app help center had a keyword search box. It was bolted onto a support platform that served dozens of enterprise tenants. Each tenant had its own configuration docs, runbooks, and internal knowledge base articles. Keyword search returned results that matched the words but missed the point. Support agents mostly ignored the box. They searched manually across a dozen browser tabs instead. Average first-response time on tickets that should have been self-service kept creeping up. Escalations to engineering were turning into a bottleneck.

The team had already prototyped a retrieval-augmented answer box in a notebook. It looked great in a demo built on one tenant’s data. A single tenant’s documentation is small and internally consistent. Indexing it in one pass is straightforward. The demo said nothing about what would happen once 40 tenants’ worth of overlapping, sometimes contradictory documentation went live at the same time. Each tenant expected its own answers, and nobody else’s. That gap, between a one-tenant demo and a forty-tenant production system, is where the real risk sat.

What we ruled out, and what we built

The notebook pointed at the fastest way to ship. Take the working single-tenant index. Widen it to cover every tenant. Filter results in the application layer before they reach an agent. We ruled that out first. A shared index across 40 tenants means a query from Tenant A’s support agent can technically retrieve Tenant B’s internal notes. A UI-side filter is a downstream promise, not an architecture. For a platform holding other companies’ internal runbooks, that gap was not one to carry from a notebook into production.

We also ruled out a chat-style answer box. It would hand an agent a single generated response with no way to check it. The notebook demo had leaned this way, because a fluent paragraph reads well in a one-tenant walkthrough. In production, an agent has to trust an answer before repeating it to a customer. A confident-sounding paragraph with nothing underneath it gives them nothing to check.

What we built instead started with per-tenant index partitioning. Isolation is enforced at the retrieval layer itself, not bolted on afterward. A query from Tenant A’s support agent cannot surface Tenant B’s internal notes. The index never holds both tenants’ documents in the same searchable set to begin with.

On top of that partitioned index sits an ingestion job. It re-indexes each tenant’s knowledge base and recent resolved tickets on a schedule. The notebook demo had run against one static snapshot, loaded once and never refreshed. A support platform’s documentation does not sit still. Runbooks get rewritten. Tickets get resolved. Configuration pages drift week to week. A scheduled re-index means an agent’s answer reflects a tenant’s current docs, not whatever was true the day someone first loaded the data.

In place of the chat-style black box, agents see a ranked-answer UI built into the existing support console. It shows the top candidate passages, each with a source link attached, instead of one generated paragraph asserting an answer. An agent can open the source and check it before repeating it to a customer. That single design choice did more to build trust in the tool than any accuracy number could have on its own.

Underneath all of it runs an eval suite. It is built from a sample of real historical tickets with known-good resolutions. It runs in CI against every change to the retrieval or ranking logic, the same way a test suite gates a code change before it merges. That is what let the team keep tuning the ranking logic with confidence, once the pipeline was live across 40 tenants instead of one.

The outcome

The pipeline went live to all tenants over a phased rollout. It started with a handful of accounts before the full base. Escalation rates and first-response times both improved measurably within the first 60 days. The metric the team cared about most going in, though, was zero cross-tenant data exposure incidents across the rollout. That is evidence the isolation model held up under real production load, not just in a one-tenant demo.

For the support lead, the win was trust rather than raw speed. Agents had learned to ignore the old keyword box, because it wasted their time on results that looked plausible and weren’t. A ranked-answer UI with a source link on every passage gave them something to check before they repeated it to a customer. That is what got the tool used on every ticket, instead of bypassed on the hard ones.

For engineering, the value sat in the eval suiteAn eval suite is a versioned set of test cases that measures whether outputs are good. running in CI on every change. Before it existed, a ranking tweak was a change nobody could measure until an agent complained about a bad result. After it existed, every change to the retrieval or ranking logic had to clear the same bar before it could ship. That bar was built from real resolved tickets. It turned the team’s confidence in its own search tool from a feeling into a number they could point to.

For security, the isolation model was the entire point of building a partitioned index instead of widening the notebook’s shared one. Zero cross-tenant exposure incidents across a rollout spanning more than 40 tenants is the result that mattered most. It is the one failure mode a support platform holding other companies’ internal documentation cannot afford even once.