Your RAG Demo Lied to You

A RAG demo that looks flawless on ten clean PDFs lies to you. Its builder unconsciously built a favorable case: clean text, obvious structure, questions paraphrased from the source. Real corpora are scanned PDFs, tables split across chunks, and users who ask in their own words. Retrieval breaks at chunking, precision/recall tradeoffs, messy documents, and vocabulary mismatch. Evaluate retrieval as its own component, against your real corpus, first.

Every retrieval-augmented demo we’ve seen follows the script: ten to twenty tidy documents, and a few test questions from someone who already knows the answers. The result looks close to perfect. It’s not a fake result. The system does retrieve the right chunk and answer correctly, for those documents, on those questions. It’s also not evidence the system will work on a real corpus. The gap between the two is where most RAG projects stall after launch.

Where does retrieval break on a real corpus?

Chunking throws away the structure that made the demo work. Splitting a document into fixed-size chunks is easy to implement. A typical starting point is 500 to 1,000 tokens per chunk. It is also reliably wrong for anything with real structure. A table gets sliced across chunk boundaries, so no single chunk contains a complete row. A numbered list loses its heading. A clause loses the definition it depends on two paragraphs up. The chunk that gets embedded and retrieved often misses the exact context that made its sentence meaningful. The system has no way to know that happened.

Precision and recall trade off, and most teams only tune for one. A retriever that returns more chunks per query catches more of the answers that matter. That is higher recall. It also buries the model in irrelevant context, which raises cost and the odds of a wrong synthesis. That is lower precision. A retriever that returns fewer, tighter chunks is precise when it’s right. It is silently wrong when the answer lives in a chunk that didn’t make the cut. Tuning one metric without measuring the other lets a team improve what it watches while what it ignores gets worse.

Messy source documents produce confidently wrong extraction. A scanned contract run through OCR. A PDF where the text layer and the visual layout disagree. A table exported to a format that loses its column alignment. Each of these produces text that looks fine to a human skimming it, yet is garbled enough to break a chunker or an extractor. This isn’t a simple retrieval miss. It’s retrieval that returns the wrong content with total confidence, a worse failure because nothing in the output signals that something went wrong.

Real questions often don’t match the document’s vocabulary. A user who asks to get their money back needs a clause about refund eligibility. A purely semantic embedding match can miss that gap as easily as a keyword search can. The demo’s questions were often written by paraphrasing the source text. That flatters whatever retrieval approach was used, and it tells you nothing about how the system handles a real user’s phrasing.

How do you evaluate retrieval quality specifically?

The fix starts with treating retrieval as its own component, with its own eval suite. That is separate from asking whether the final answer sounds right. A generation-only eval can pass on top of retrieval that fetched the wrong document entirely. The model can still write a fluent, plausible-sounding answer. A capable model will confidently synthesize an answer from whatever context it’s given, correct or not. That failure mode is the most dangerous one in RAG: it looks like success right up until someone checks the source.

A real retrieval eval needs a set of queries with known correct source documents, not just correct answers. Pull them from your actual corpus, including the messy parts: the scanned contract, the table-heavy spreadsheet, the slide deck. A useful starting size is 50 to 100 query and document pairs, enough to catch a regression without becoming its own maintenance burden. Measure recall and precision separately, because a fix to one can hurt the other. Recall asks whether the right document made it into the retrieved set at all. Precision asks how much of what got retrieved was relevant. Test with real user phrasing too, not paraphrases of the source text. Pull that phrasing from actual query logs once the feature has usage, or from people who didn’t write the source documents if it doesn’t yet.

How does this fit into a client engagement?

When we build retrieval into a feature, we build the eval suiteAn eval suite is a versioned set of test cases that measures whether outputs are good. against the client’s actual documents before we tune anything. That means the messy PDFs, the inconsistent formatting, the real question patterns. It draws on the same ten-to-twenty-document sample size a demo would use, but we choose it for messiness instead of polish. It’s slower than shipping the version that impresses in a meeting. It’s also the difference between a retrieval system that degrades gracefully as the corpus grows and one that looked perfect on day one. The second kind gets worse every week, and nobody notices until a user asks something the demo never covered.

These failures have their own diagnostic entries: stale indexes, split chunks, and similarity mistaken for relevance. See stale index serves deleted content, chunk boundary splits the answer, and similarity is not relevance in the field guide.