How do you chunk a document without splitting the answer in half?

What problem does this solve?#

Fixed-size chunkingChunking splits source documents into smaller passages before embedding, so retrieval returns focused text. splits the text every N tokens, optionally with overlap. It is the default, because it is simple to implement and works well on plain prose. It has no concept of a table or a heading hierarchy. A chunk boundary lands wherever the token count runs out. That spot is rarely a row boundary or a clause boundary. The chunker slices a table mid-row. A procedure loses the trigger condition that appears two lines above its steps.

Each half of a split chunk is accurate for what it contains but incomplete on its own. It embeds as a weak, ambiguous match for a query that the full passage would have answered clearly. The ranker then does its job and drops the low-scoring fragments. The failure is upstream of ranking. It is baked in at ingest time, before retrieval ever runs.

How does it work?#

The chunker first parses the document structure, which includes headings and tables. It also parses lists. It falls back to a token-count split only within a leaf node, such as a long paragraph inside one heading. A leaf node has no finer structure left to respect. The chunker splits a table into whole rows and repeats the header row in each chunk body. A small table stays as one chunk. A numbered list keeps its trigger condition and step number together.

Every chunk carries its ancestor heading path, for example "Refunds > Partial refunds > Time limits". The chunker prepends the path into the embedded text. It also stores the path as metadata. A chunk that reads "Requests must be submitted within 14 days" then still identifies its topic out of context. Metadata helps a human who reads the trace. Context in the text changes what the embedding represents.

A second layer retrieves the small, precise chunk for matching. It expands to the full parent section when it builds the model context. This recovers cases where structural splitting still leaves a chunk narrower than the full answer. The cost is a larger context per call.

What are the moving parts?#

1. Parse structure before splitting

   Read headings and tables first. Apply a fixed token count only when no structure remains.

2. Never split a row or a step

   A table row stays with its header. A list item stays with its identifying context.

3. Carry the ancestor heading path into the chunk text

   Prepend the heading hierarchy into the embedded body itself. The embedding then sees it.

4. Split by token count only inside a leaf node

   A long unstructured paragraph within one heading is the only place where a fixed-size split still applies.

5. Retrieve the child chunk, feed the parent section

   Match on the precise chunk. Expand to the full parent section when you build the text the model reads.

6. Maintain a regression fixture of known answers

   Pair known answers with the query that should retrieve them. Check pass or fail after every chunking change.

When does it fit, and when doesn't it?#

What does it cost?#

How do you know it's working?#

• Recall@k on a fixed set of known answer and query pairs, tracked before and after every chunking change
• Rate of 'answer confirmed in source, absent from every retrieved chunk' incidents
• Average and p95 chunk size, watched for a fixed-size chunker that returns through a fallback path
• Share of retrieved chunks that come from a table or list, compared with plain prose, to catch regressions concentrated there

What failure modes does this prevent?#