Grounded answering over your own documents
Answer questions from your own documents, with a citation, instead of from whatever the model learned during training.
“We want a search box or chatbot that answers from our own docs and can show where the answer came from.”
What does the system look like?#
How does a request move through it?#
- Ingest on change, not on a timer
Source documents enter through a change-event subscription rather than a scheduled recrawl, so an edit or delete reaches the index in minutes.
- Chunk on structure
Headings, tables, and list items set the chunk boundaries, never a fixed token count, so a chunk is never split apart from the header or step it needs to make sense.
- Embed and index
Each chunk carries its ancestor heading path into the embedded text itself, so it still identifies what it is about once retrieved on its own.
- Retrieve a wide set, then rerank it down
A cheap first pass pulls 50-100 candidates favoring recall; a cross-encoder reranks them and applies a relevance floor, so the system can answer "nothing here" instead of forcing a weak match to the top.
- Generate only from floor-cleared context
The model answers strictly from the reranked, floor-passing chunks and cites which one each claim came from, never from chunks the floor rejected.
- Trace, eval, and feed incidents back
Every run is traced end to end; a wrong or unfindable answer becomes a new case in the eval set and, if it traces to a stale or badly split chunk, a fix in ingest or chunking, ahead of a one-off prompt patch.
What are the pieces, and what breaks without each one?#
Change-driven ingest
Subscribe to the source system's real change events and re-embed or tombstone within minutes of an edit or delete.
Breaks without it: The index quietly drifts from the source and starts confidently citing a policy that was deleted months ago.
Structure-aware chunker
Split on headings, tables, and list items, and carry the ancestor heading path into the embedded text.
Breaks without it: A table row loses its header, a step loses its trigger condition, and the answer that is plainly in the document never makes it into any single retrieved chunk.
Retriever + reranker
Pull a wide, high-recall candidate set, then rerank it with a model that scores query and chunk jointly and can return "no confident match".
Breaks without it: A topically similar but factually wrong document quietly outranks the correct one, especially on identifier or date-shaped queries.
Grounded generator
Answer only from the chunks that cleared the relevance floor, and cite which one supports each claim.
Breaks without it: The model falls back on whatever it learned during training the moment retrieval comes up short, and a reader has no way to tell the difference.
Trace store
Log the exact query, retrieved chunks, and generated answer for every run under one trace ID.
Breaks without it: A user-reported wrong answer becomes a guess instead of a five-minute lookup of exactly what the system saw.
Eval gate
Run a labelled set of real questions, including identifier and date-shaped ones scored separately from topical ones, on every ingest or model change.
Breaks without it: A chunkingChunking splits source documents into smaller passages before embedding, so retrieval returns focused text. or model change quietly regresses on exactly the query shape nobody happened to re-test by hand.
Where does this need a decision, not a default?#
| Decision | Default choice | Why |
|---|---|---|
| Vector store | pgvector if you already run Postgres and the corpus is in the low millions of vectors | It removes an entire piece of infrastructure to operate. A dedicated vector database earns its keep once scale, filtering complexity, or query latency outgrow what an extension on your primary database can deliver. |
| Embedding model | A hosted embedding API to start, self-hosted only once volume justifies the ops cost | Embedding calls are cheap and easy to swap behind an abstraction layer; the harder decision is the chunking and index design around them, not which model produces the vectors. |
| Reranker | A hosted cross-encoder API, added once single-stage retrieval shows a measured precision problem | Adding a reranker before you have evidence single-stage search is actually the bottleneck is a cost and latency tax with nothing to show for it. |
| Chunking strategy | Structure-aware chunking from day one, not a fixed-size splitter with a plan to fix it later | Retrofitting structure-aware chunking after launch means re-embedding the whole corpus and re-validating every downstream eval case against the new boundaries. |
What actually drives the bill?#
- Generation tokens per answered query
- Reranker calls on the retrieved candidate set
- Embedding calls at ingest time, spiking on a bulk re-index
Dominant cost Generation tokens per query once the corpus and ingest pipeline stabilize, because every answered question pays for a model call, while a given document is embedded only once.
The lever A model cascade on generation (cheap model first, escalate on low confidence) plus stable-prefix prompt caching on the system instructions and retrieved-context template.
Model your own numbers with the AI Cost Calculator →How do you know it actually works?#
Recall@k, scored separately from generation quality
Confirms the retriever hands the generator the right chunk before generation quality can hide the gap.
Identifier and date-query set, reported apart from the topical average
Catches the exact query shape cosine similarity handles worst.
Groundedness / attribution rate
Checks that every claim in the answer traces to the specific chunk cited for it.
Blocks a chunking, model, or retrieval change that regresses a known-good case.
What will your reviewers ask about this?#
Logging & retention with PII redaction
Prompt and completion logs get a stated retention period and a redaction pass before they reach a third-party observabilityObservability captures traces of every prompt, retrieval, tool call, and response for debugging. tool.
Data-subject deletion path covering trace and eval stores
A deletion request has to reach the trace store and eval set, alongside the primary index.
Subprocessor visibility on the model and embedding providers
Whoever reviews vendor risk needs the full chain, beyond the provider named on the contract.
Which patterns and failure modes tie in?#
How long does a first version actually take?#
4-8 weeks to a pilot that answers real questions against a live corpus; add 2-4 weeks to build change-data-capture ingest and a real eval set, the difference between a pilot and a one-time batch load behind a demo script.
When is this the wrong shape?#
- Your corpus is small and stable enough that a person could just read it, or a decent keyword search bar already finds the answer.
- You need the system to take actions on your behalf, beyond answering questions. That is the agentic workflow runner, not this.
- Nobody on your side will own keeping the eval set current after launch. A grounded-answering system without a maintained eval set degrades quietly and nobody notices until a customer does.
Sources
- arXiv - Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
The paper that named retrieval-augmented generation as a technique. Retrieved 2026-08-24.
- arXiv - Passage Re-ranking with BERT
The paper that established cross-encoder scoring as the second-stage reranking method our retrieve-then-rerank pattern applies. Retrieved 2026-09-17.
A Ship Audit checks which of these an existing or planned system actually needs, against what is most likely to break first.