Your RAG Demo Lied to You
Retrieval that looks flawless on ten clean PDFs falls apart on a real corpus. Here's why, and what evaluating retrieval quality actually requires.
Read → Topics
Retrieval & RAG
Retrieval-augmented generation looks simple in a demo and breaks on real corpora — messy PDFs, thin chunking, and stale embeddings all masquerade as model problems. This is our work on making retrieval actually hold up in production.
Insights
Case studies
Retrieval Pipeline That Actually Cut Support Load
A mid-market SaaS support platform replaced a keyword search widget with a tenant-isolated retrieval pipeline, cutting escalations and first-response time without a headcount increase.
Read →Failure modes
Stale index serves deleted content
Your retrieval index was built once at ingest and never told the source changed. When a document is edited or deleted, nothing re-embeds the new version or tombstones the old chunk, so the stale vector keeps scoring well and keeps getting served — confidently, and with no signal to the reader that it is out of date.
Read →Chunk boundary splits the answer
The answer exists in the source, but a fixed-size chunker cut it in half at ingest time — a table row split from its header, a procedure split from its trigger condition. Each half scores weakly on its own, the ranker drops both, and retrieval reports nothing when the document plainly contains the answer.
Read →Similarity is not relevance
Cosine similarity rewards topical resemblance, not correctness — it can rank a document about the wrong product, the wrong date, or the negated version of a claim above the one that actually answers the query, because embeddings represent "about the same thing" far more reliably than they represent identifiers, negation, or numbers.
Read →Patterns
Structure-aware chunking
Chunk boundaries follow the document's own structure — headings, table rows, list items, section boundaries — instead of a fixed token count, and each chunk carries its parent heading or identifying context in its own body. A table row is never separated from its header; a step is never separated from the procedure it belongs to. The chunk that gets embedded is always a complete unit of meaning, not an arbitrary slice.
Read →Retrieve-then-rerank
A cheap, high-recall first pass — vector search, optionally fused with keyword search — pulls a wide candidate set of 50 to 100 documents likely to contain the right answer somewhere. A precision reranker, usually a cross-encoder that scores the query and each candidate jointly, then re-sorts that set and applies a relevance floor allowed to return nothing rather than force a weak match to the top.
Read →Change-data-capture ingest
The ingest pipeline subscribes to the actual change events of the source system — a webhook, a CMS publish hook, a database trigger — instead of re-crawling on a fixed schedule. An edit triggers a re-embed within minutes; a delete writes a tombstone that excludes the old chunks from retrieval immediately, before a full re-index even runs.
Read →