# How do you chunk a document without splitting the answer in half? Source: https://customlabs.io/patterns/structure-aware-chunking/ Updated: 2026-09-17 Retrieval # How do you chunk a document without splitting the answer in half? Updated September 17, 2026 · First published July 28, 2026 · 5 min read · Intent Chunk boundaries follow the document's own structure: headings, table rows, list items, section boundaries, instead of a fixed token count. Each chunk carries its parent heading or identifying context in its own body. A table row is never separated from its header, and a step is never separated from the procedure it belongs to. The chunk that gets embedded is always a complete unit of meaning, not an arbitrary slice. Also known as structural chunking, hierarchy-aware splitting, heading-anchored chunks Problem ## What problem does this solve? Fixed-size [chunking](https://customlabs.io/glossary/chunking/) splits every N tokens, optionally with overlap. It's the default, because it's trivial to implement and works fine on plain prose. It has no concept of a table, a numbered list, or a heading hierarchy. A chunk boundary lands wherever the token count runs out, which is rarely a row boundary or a clause boundary. A table gets sliced mid-row. A procedure loses the trigger condition stated two lines above its steps. Each half of a split chunk is individually accurate to what it contains, but incomplete on its own. It embeds as a weak, ambiguous match for a query the full passage would have answered clearly. The ranker then does exactly its job and drops the low-scoring fragments. The failure isn't a ranking bug. It's upstream of ranking, baked in at ingest time before retrieval ever runs. Mechanics ## How does it work? The chunker parses document structure first: headings, tables, ordered and unordered lists. It only falls back to a token-count split within a leaf node, such as a long paragraph inside one heading, that has no finer structure left to respect. A table is chunked as whole rows, with the header row repeated into each chunk's body, or kept as one chunk if it's small enough. A numbered list keeps its trigger condition and step number together. Every chunk carries its ancestor heading path, for example "Refunds > Partial refunds > Time limits," prepended directly into the embedded text, beyond being stored as metadata. So a chunk reading "Requests must be submitted within 14 days" still identifies what it's about, even out of context. This is the difference between metadata that helps a human reading the trace, and context that actually changes what the embedding represents. A second layer retrieves the small, precise chunk for matching, but expands to its full parent section when constructing the model's context. It recovers cases where structural splitting still leaves a chunk narrower than the full answer, at the cost of a slightly larger context per call. Shape ## What are the moving parts? - Parse structure before splitting Read headings, tables, and lists first, rather than defaulting straight to a fixed token count. - Never let a boundary cross a row, step, or clause Table rows, list items, and their identifying context stay together in one chunk by construction. - Carry the ancestor heading path into the chunk text Prepend the heading hierarchy into the embedded body itself, where the embedding actually sees it. - Fall back to token-count splitting only inside a leaf node A long unstructured paragraph within one heading is the only place a fixed-size split is still appropriate. - Retrieve the child chunk, feed the parent section Match on the precise chunk, but expand to the full parent section when constructing what the model actually reads. - Maintain a regression fixture of known answers Pair known answers with the query that should retrieve them, and check pass/fail after every chunking change. Fit ## When does it fit, and when doesn't it? ### Use it when - The corpus has real structure: tables, numbered procedures, nested headings, rather than flowing prose - Users report an answer 'is definitely in there' but retrieval keeps missing it - The corpus changes often enough that a one-time hand-tuned chunking pass won't stay correct - You are already indexing structured formats (Markdown, HTML, DOCX with real headings) that a parser can read ### Don't use it when - The corpus is genuinely unstructured free text (chat transcripts, raw OCR output) with no headings or tables to parse. There is no structure to be aware of - Document volume is small enough that hand-curated chunks are cheaper to maintain than a structural parser - The source format strips structure on the way in (plain-text exports with headings flattened). Fix the ingestion format first, or this pattern has nothing to key off Trade-offs ## What does it actually cost? Gain A chunk boundary never severs a row, step, or clause from what identifies it Cost A real structural parser per source format (Markdown, HTML, DOCX, PDF) is more implementation work than one universal token splitter Gain Ancestor-heading context makes a chunk self-identifying even out of context Cost Chunks get slightly larger on average (repeated heading path per chunk), raising index size and per-call token cost a little Gain Retrieve-child-feed-parent recovers edge cases structural splitting alone still misses Cost Parent expansion means more tokens land in the model's context per retrieved match, which competes with your context budget Gain A regression fixture catches chunking regressions before they ship Cost Someone has to build and maintain that fixture. It does not exist for free just because you adopted structural chunking Signals ## How do you know it's working? - Recall@k on a fixed set of known answer/query pairs, tracked before and after every chunking change - Rate of 'answer confirmed in source, absent from every retrieved chunk' incidents - Average and p95 chunk size, watched for a fixed-size chunker sneaking back in via a fallback path - Share of retrieved chunks that come from a table or list vs. plain prose, to catch regressions concentrated there Prevents ## What failure modes does this prevent? [Chunk boundary splits the answer](https://customlabs.io/failure-modes/chunk-boundary-splits-the-answer/) ### Sources - [arXiv - Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401) The paper that named retrieval-augmented generation as a technique. Retrieved 2026-08-24. - [arXiv - Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172) The paper measuring how a fact's position in a context window affects recall. Retrieved 2026-08-24. In the Handbook [02 Design](https://customlabs.io/handbook/design/) Related patterns [Retrieve-then-rerank](https://customlabs.io/patterns/retrieve-then-rerank/)[Change-data-capture ingest](https://customlabs.io/patterns/change-data-capture-ingest/) Topics [Retrieval & RAG](https://customlabs.io/topics/retrieval-rag/) Related reading [Glossary: Chunking](https://customlabs.io/glossary/chunking/)[Insight: Your RAG demo lied to you](https://customlabs.io/insights/your-rag-demo-lied/) Not sure this is the right pattern? A Ship Audit checks which of these patterns your system actually needs, prioritized against what's most likely to break first. [Book a Ship Audit →](https://customlabs.io/diagnostic/ship-audit/)