Why can't retrieval find an answer that is definitely in the document?

What does this look like in production?

• A user pastes the exact sentence from the source document and retrieval still fails to surface the chunk containing it
• The answer is present when you open the document manually but absent from every retrieved chunk in the trace
• Failures cluster around tables, numbered procedures, and multi-clause definitions rather than plain prose
• Increasing top-k retrieved chunks barely moves the failure rate, because more of the same badly-cut chunks doesn't fix a chunk that's missing half its meaning

Why does it happen?

Fixed-size chunking — split every N tokens, optionally with a fixed overlap — is the default in most RAG starter kits because it's trivial to implement and works fine on plain paragraphs. It has no concept of document structure: a table gets sliced wherever the token count runs out, which is rarely a row boundary, so no single chunk contains a complete row and its header together. A numbered procedure loses the step number or the trigger condition it depends on two lines up. The embedding for each half is technically accurate to what's in it — but what's in it is incomplete, so it scores as a weak, ambiguous match for a query the full passage would have answered clearly.

The ranker then does exactly what it's supposed to do: it drops low-scoring chunks. The failure isn't a ranking bug, it's upstream of ranking — the chunk it's scoring was never a complete unit of meaning to begin with.

How do you confirm it's this?

• On a known failing query, grep the raw source corpus (not the index) for the expected answer string and confirm it exists verbatim
• Then check every retrieved chunk for that query and confirm none of them contains the full answer string — if it's split across two adjacent chunks, that's the defect
• Build a small regression fixture: 10-20 known answers paired with the query that should retrieve them, and check pass/fail after every chunking change
• Visually inspect chunk boundaries around tables and numbered lists specifically — this is where fixed-size splitting fails first and worst

How do you fix it?

1. Split on structure, not token count

   Chunk on headings, table boundaries, and list items so a chunk boundary never lands inside a row, a step, or a clause — the chunker should respect the document's own structure before it respects a token budget.

2. Add overlap as a second line of defense

   Even with structure-aware splitting, a modest token overlap between adjacent chunks catches boundary cases that structural rules miss, at the cost of some index size.

3. Retrieve the child, feed the parent

   Index small, precise chunks for matching (better retrieval precision) but expand to the full parent section or document when feeding the model, so the model always sees complete context even when the match came from a fragment.

4. Keep a regression fixture per known answer

   Every time a real "the answer is in there and retrieval missed it" incident happens, add it to a fixed test set that runs before any chunking or embedding-model change ships.

What this doesn't cover

This is specifically a fragmentation failure — if the chunk containing the full answer is retrieved and the model still gets it wrong, the defect is in generation or ranking, not chunking. Verify the complete answer is genuinely absent from every retrieved chunk before rebuilding your chunking strategy.