Why can't retrieval find an answer that is definitely in the document?

What does this look like in production?

• A user pastes the exact sentence from the source document and retrieval still fails to surface the chunk containing it
• The answer is present when you open the document manually but absent from every retrieved chunk in the trace
• Failures cluster around tables, numbered procedures, and multi-clause definitions rather than plain prose
• Raising top-k barely moves the failure rate. More badly cut chunks do not fix a chunk that is missing half its meaning

Why does it happen?

Fixed-size chunkingChunking splits source documents into smaller passages before embedding, so retrieval returns focused text. splits every N tokens, optionally with a fixed overlap. It's the default in most RAGRetrieval-Augmented Generation (RAG) retrieves relevant passages at query time and feeds them into an LLM's context. starter kits because it's trivial to implement and works fine on plain paragraphs. It has no concept of document structure. A table gets sliced where the token count runs out, which is rarely a row boundary. No single chunk then holds a complete row and its header. A numbered procedure loses the step number or the trigger condition it depends on two lines up. The embedding for each half is technically accurate to what's in it. But what's in it is incomplete, so it scores as a weak, ambiguous match for a query the full passage would have answered clearly.

The ranker then does exactly what it's supposed to do: it drops low-scoring chunks. The failure isn't a ranking bug. It's upstream of ranking: the chunk it's scoring was never a complete unit of meaning to begin with.

How do you confirm it's this?

• On a known failing query, grep the raw source corpus (not the index) for the expected answer string and confirm it exists verbatim
• Then check every retrieved chunk for that query and confirm none of them contains the full answer string. If it's split across two adjacent chunks, that's the defect
• Build a small regression fixture: 10-20 known answers paired with the query that should retrieve them, and check pass/fail after every chunking change
• Visually inspect chunk boundaries around tables and numbered lists specifically. This is where fixed-size splitting fails first and worst

How do you fix it?

1. Split on structure, not token count

   Chunk on headings, table boundaries, and list items so a chunk boundary never lands inside a row, a step, or a clause. The chunker should respect the document's own structure before it respects a token budget.

2. Add overlap as a second line of defense

   Even with structure-aware splitting, a modest token overlap between adjacent chunks catches boundary cases that structural rules miss, at the cost of some index size.

3. Retrieve the child, feed the parent

   Index small, precise chunks for matching, which improves retrieval precision. When you feed the model, expand to the full parent section or document. The model then sees complete context even when the match came from a fragment.

4. Keep a regression fixture per known answer

   Each time a real incident happens where the answer is in the corpus and retrieval missed it, add it to a fixed test set. Run that set before any chunking or embedding-model change ships.

What this doesn't cover

This is specifically a fragmentation failure. If the chunk containing the full answer is retrieved and the model still gets it wrong, the defect is in generation or ranking, not chunking. Verify the complete answer is genuinely absent from every retrieved chunk before rebuilding your chunking strategy.