Retrieval Pipeline That Actually Cut Support Load

The problem

Their in-app help center had a keyword search box bolted onto a support platform serving dozens of enterprise tenants, each with its own configuration docs, runbooks, and internal knowledge base articles. Keyword search returned technically-matching but practically-useless results, so support agents mostly ignored it and searched manually across a dozen browser tabs. Average first-response time on tickets that should have been self-service was creeping up, and escalations to engineering were becoming a bottleneck.

The team had prototyped a retrieval-augmented answer box in a notebook. It looked great in a demo with one tenant’s data loaded. It said nothing about what would happen with 40 tenants’ worth of overlapping, sometimes contradictory documentation live at once.

What we built

• A retrieval pipeline with per-tenant index partitioning, so a query from Tenant A’s support agent can never surface Tenant B’s internal notes — enforced at the retrieval layer, not just the UI.
• An ingestion job that re-indexes each tenant’s knowledge base and recent resolved tickets on a schedule, so answers reflect the current state of their docs rather than a stale snapshot.
• A ranked-answer UI in the existing support console: top candidate passages with source links, not a black-box chat response, so agents can verify before they act.
• An eval suite built from a sample of real historical tickets with known-good resolutions, run in CI against every change to the retrieval or ranking logic.

The outcome

The pipeline went live to all tenants over a phased rollout, starting with a handful of accounts before the full base. Escalation rates and first-response times both improved measurably within the first 60 days. The metric the team cared about most going in, though, was zero cross-tenant data exposure incidents across the rollout, evidence the isolation model held up under real production load.