How do you chunk a document without splitting the answer in half?
Chunk boundaries follow the document's own structure — headings, table rows, list items, section boundaries — instead of a fixed token count, and each chunk carries its parent heading or identifying context in its own body. A table row is never separated from its header; a step is never separated from the procedure it belongs to. The chunk that gets embedded is always a complete unit of meaning, not an arbitrary slice.
Also known as structural chunking, hierarchy-aware splitting, heading-anchored chunks
What problem does this solve?
Fixed-size chunking — split every N tokens, optionally with overlap — is the default because it's trivial to implement and works fine on plain prose. It has no concept of a table, a numbered list, or a heading hierarchy: a chunk boundary lands wherever the token count runs out, which is rarely a row boundary or a clause boundary. A table gets sliced mid-row; a procedure loses the trigger condition stated two lines above its steps.
Each half of a split chunk is individually accurate to what it contains but incomplete on its own, so it embeds as a weak, ambiguous match for a query the full passage would have answered clearly. The ranker then does exactly its job and drops the low-scoring fragments — the failure isn't a ranking bug, it's upstream of ranking, baked in at ingest time before retrieval ever runs.
How does it work?
The chunker parses document structure first — headings, tables, ordered and unordered lists — and only falls back to a token-count split within a leaf node, such as a long paragraph inside one heading, that has no finer structure left to respect. A table is chunked as whole rows with the header row repeated into each chunk's body, or kept as one chunk if it's small enough; a numbered list keeps its trigger condition and step number together.
Every chunk carries its ancestor heading path — for example "Refunds > Partial refunds > Time limits" — prepended into the embedded text, not just stored as metadata, so a chunk reading "Requests must be submitted within 14 days" still identifies what it's about even out of context. This is the difference between metadata that helps a human reading the trace and context that actually changes what the embedding represents.
A second layer — retrieve the small, precise chunk for matching, but expand to its full parent section when constructing the model's context — recovers cases where structural splitting still leaves a chunk narrower than the full answer, at the cost of a slightly larger context per call.
What are the moving parts?
- Parse structure before splitting
Read headings, tables, and lists first, rather than defaulting straight to a fixed token count.
- Never let a boundary cross a row, step, or clause
Table rows, list items, and their identifying context stay together in one chunk by construction.
- Carry the ancestor heading path into the chunk text
Prepend the heading hierarchy into the embedded body itself, not just into a metadata field the embedding never sees.
- Fall back to token-count splitting only inside a leaf node
A long unstructured paragraph within one heading is the only place a fixed-size split is still appropriate.
- Retrieve the child chunk, feed the parent section
Match on the precise chunk, but expand to the full parent section when constructing what the model actually reads.
- Maintain a regression fixture of known answers
Pair known answers with the query that should retrieve them, and check pass/fail after every chunking change.
When does it fit, and when doesn't it?
Use it when
- The corpus has real structure — tables, numbered procedures, nested headings — not just flowing prose
- Users report an answer 'is definitely in there' but retrieval keeps missing it
- The corpus changes often enough that a one-time hand-tuned chunking pass won't stay correct
- You are already indexing structured formats (Markdown, HTML, DOCX with real headings) that a parser can read
Don't use it when
- The corpus is genuinely unstructured free text (chat transcripts, raw OCR output) with no headings or tables to parse — there is no structure to be aware of
- Document volume is small enough that hand-curated chunks are cheaper to maintain than a structural parser
- The source format strips structure on the way in (plain-text exports with headings flattened) — fix the ingestion format first, or this pattern has nothing to key off
What does it actually cost?
A chunk boundary never severs a row, step, or clause from what identifies it
A real structural parser per source format (Markdown, HTML, DOCX, PDF) is more implementation work than one universal token splitter
Ancestor-heading context makes a chunk self-identifying even out of context
Chunks get slightly larger on average (repeated heading path per chunk), raising index size and per-call token cost a little
Retrieve-child-feed-parent recovers edge cases structural splitting alone still misses
Parent expansion means more tokens land in the model's context per retrieved match, which competes with your context budget
A regression fixture catches chunking regressions before they ship
Someone has to build and maintain that fixture — it does not exist for free just because you adopted structural chunking
How do you know it's working?
- Recall@k on a fixed set of known answer/query pairs, tracked before and after every chunking change
- Rate of 'answer confirmed in source, absent from every retrieved chunk' incidents
- Average and p95 chunk size, watched for a fixed-size chunker sneaking back in via a fallback path
- Share of retrieved chunks that come from a table or list vs. plain prose, to catch regressions concentrated there
What failure modes does this prevent?
A Ship Audit checks which of these patterns your system actually needs, prioritized against what's most likely to break first.