Your RAG Demo Lied to You

A RAG demo that looks flawless on ten clean PDFs lies because its builder unconsciously fed it a favorable case: clean text, obvious structure, questions paraphrased from the source. Real corpora are scanned PDFs, tables split across chunks, and users who ask in their own words. Retrieval breaks at chunking, precision/recall tradeoffs, messy documents, and vocabulary mismatch. Evaluate retrieval as its own component, against your real corpus, first.

Every retrieval-augmented demo we’ve seen follows the same script: ten to twenty well-formatted documents, a handful of test questions written by someone who already knows the answers are in there, and a result that looks close to perfect. It’s not a fake result. The system genuinely does retrieve the right chunk and answer correctly, for those documents, on those questions. It’s also not evidence the system will work on a real corpus, and the gap between the two is where most RAG projects quietly stall after launch.

Where does retrieval actually break on a real corpus?

Chunking throws away the structure that made the demo work. Splitting a document into fixed-size chunks (a typical starting point is 500–1,000 tokens) is easy to implement and reliably wrong for anything with real structure: a table gets sliced across chunk boundaries so no single chunk contains a complete row, a numbered list loses its heading, a clause loses the definition it depends on two paragraphs up. The chunk that gets embedded and retrieved is often missing the exact context that made the sentence inside it meaningful, and the system has no way to know that happened.

Precision and recall trade off, and most teams only tune for one. A retriever that returns more chunks per query catches more of the answers that matter (higher recall) but buries the model in irrelevant context that increases both cost and the odds of a wrong synthesis (lower precision). A retriever that returns fewer, tighter chunks is precise when it’s right and silently wrong when the answer lives in a chunk that didn’t make the cut. Tuning one without measuring the other is how a team improves the metric they’re watching while the metric they’re not watching gets worse.

Messy source documents produce confidently wrong extraction. A scanned contract run through OCR, a PDF where the text layer and the visual layout disagree, a table exported to a format that loses its column alignment: all of these produce text that looks fine to a human skimming it and is garbled enough to break a chunker or an extractor. This isn’t a simple retrieval miss. It’s retrieval that returns the wrong content with total confidence, a worse failure mode because nothing about the output signals that something went wrong.

Real questions often don’t match the document’s vocabulary. A user asking “can I get my money back” needs to retrieve a clause that says “refund eligibility,” and a purely semantic embedding match can miss that gap just as easily as a keyword search can. The demo’s questions were often written by paraphrasing the source text, which flatters whatever retrieval approach was used and tells you nothing about how it performs against a real user’s phrasing.

How do you evaluate retrieval quality specifically?

The fix starts with treating retrieval as its own component with its own eval suite, separate from “did the final answer sound right.” A generation-only eval can pass (the model can write a fluent, plausible-sounding answer) on top of retrieval that fetched the wrong document entirely, because a capable model will confidently synthesize an answer from whatever context it’s given, correct or not. That failure mode is the most dangerous one in RAG: it looks like success right up until someone checks the source.

A real retrieval eval needs a set of queries with known correct source documents, not just correct answers, pulled from your actual corpus, including the messy parts: the scanned contract, the table-heavy spreadsheet, the slide deck. A useful starting size is 50–100 query/document pairs, enough to catch a regression without becoming its own maintenance burden. Measure recall (did the right document make it into the retrieved set at all) and precision (how much of what got retrieved was actually relevant) separately, because a fix to one can silently hurt the other. And test with real user phrasing, not paraphrases of the source text. Ideally, pull that phrasing from actual query logs once the feature has any usage, or from people who didn’t write the source documents if it doesn’t yet.

How does this fit into a client engagement?

When we build retrieval into a feature, the eval suite for retrieval quality gets built against the client’s actual documents before we tune anything: the messy PDFs, the inconsistent formatting, the real question patterns. It’s drawn from the same ten-to-twenty-document sample size a demo would use, but chosen for messiness instead of polish, not a curated sample chosen because it demos well. It’s slower than shipping the version that impresses in a meeting. It’s also the difference between a retrieval system that degrades gracefully as the corpus grows and one that looked perfect on day one and got quietly worse every week after, with nobody noticing until a user asked something the demo never covered.

For a diagnostic breakdown of these specific failures — stale indexes, split chunks, and similarity-as-relevance — see the field guide entries on stale index serves deleted content, chunk boundary splits the answer, and similarity is not relevance.