Grounded answering over your own documents
Answer questions from your own documents and cite the source. The model does not answer from its training data.
“We want a search box or chatbot that answers from our own docs. It must show where each answer came from.”
What does the system look like?#
How does a request move through it?#
- Ingest on change
Source documents enter through a change-event subscription. Ingest does not wait for a scheduled recrawl. An edit or delete reaches the index in minutes.
- Chunk on structure
Headings, tables, and list items set the chunk boundaries. A fixed token count does not. A chunk stays with the header or step it needs.
- Embed and index
Each chunk carries its ancestor heading path into the embedded text. A retrieved chunk then still shows what it is about.
- Retrieve a wide set, then rerank it down
A cheap first pass pulls 50-100 candidates and favors recall. A cross-encoder reranks them and applies a relevance floor. The system can then answer "nothing here" and avoid a weak match.
- Generate only from floor-cleared context
The model answers only from the reranked chunks that pass the floor. It cites the chunk behind each claim. It ignores chunks the floor rejected.
- Trace, eval, and feed incidents back
Every run is traced end to end. A wrong or missing answer becomes a new case in the eval set. If a stale or badly split chunk caused it, the team fixes ingest or chunkingChunking splits source documents into smaller passages before embedding, so retrieval returns focused text. before any prompt patch.
What are the pieces, and what breaks without each one?#
Change-driven ingest
Subscribe to the change events of the source system. Re-embed or tombstone the chunk within minutes of an edit or delete.
Breaks without it: The index drifts from the source. It then cites a policy that the team deleted months ago.
Structure-aware chunker
Split on headings, tables, and list items, and carry the ancestor heading path into the embedded text.
Breaks without it: A table row loses its header. A step loses its trigger condition. The answer is in the document but in no single retrieved chunk.
Retriever + reranker
Pull a wide candidate set with high recall. Rerank it with a model that scores query and chunk together. The model can return "no confident match".
Breaks without it: A document on the same topic but with wrong facts outranks the correct one. This happens most on identifier or date queries.
Grounded generator
Answer only from the chunks that cleared the relevance floor, and cite which one supports each claim.
Breaks without it: When retrieval comes up short, the model falls back on its training data. A reader cannot see the difference.
Trace store
Log the exact query, retrieved chunks, and generated answer for every run under one trace ID.
Breaks without it: A user reports a wrong answer and the team must guess. With a trace, the lookup of what the system saw takes five minutes.
Eval gate
Run a labelled set of real questions on every ingest or model change. Score identifier and date questions apart from topical ones.
Breaks without it: A chunking or model change makes one query shape worse. Nobody re-tests that shape by hand.
Which choices need a decision?#
| Decision | Default choice | Why |
|---|---|---|
| Vector store | pgvector if you already run Postgres and the corpus is in the low millions of vectors | It removes one piece of infrastructure to operate. A dedicated vector database pays off when scale, filtering needs, or query latency outgrow an extension on your primary database. |
| Embedding model | A hosted embedding API to start, self-hosted only once volume justifies the ops cost | Embedding calls are cheap and easy to swap behind an abstraction layer. The chunking and index design is the harder decision. The choice of vector model matters less. |
| Reranker | A hosted cross-encoder API, added once single-stage retrieval shows a measured precision problem | A reranker added without evidence that single-stage search is the bottleneck adds cost and latency. It returns no gain. |
| Chunking strategy | Structure-aware chunking from day one. Skip the fixed-size splitter that you plan to fix later | Retrofitting structure-aware chunking after launch means re-embedding the whole corpus. It also means re-validating every downstream eval case against the new boundaries. |
What drives the bill?#
- Generation tokens per answered query
- Reranker calls on the retrieved candidate set
- Embedding calls at ingest time, with a spike on a bulk re-index
Dominant cost Generation tokens per query, once the corpus and ingest pipeline stabilize. Every answered question pays for a model call. A document is embedded only once.
The lever A model cascade on generation sends the cheap model first and escalates on low confidence. Stable-prefix prompt caching covers the system instructions and the retrieved-context template.
Model your own numbers with the AI Cost Calculator →How do you know it works?#
Recall@k, scored separately from generation quality
Confirms the retriever hands the generator the right chunk. Good generation cannot hide a retrieval gap.
Identifier and date-query set, reported apart from the topical average
Catches the query shape that cosine similarity handles worst.
Groundedness / attribution rate
Checks that every claim in the answer traces to the specific chunk cited for it.
Blocks a chunking, model, or retrieval change that regresses a known-good case.
What will your reviewers ask about this?#
Logging & retention with PII redaction
Prompt and completion logs get a stated retention period. A redaction pass runs before logs reach a third-party observabilityObservability captures traces of every prompt, retrieval, tool call, and response for debugging. tool.
Data-subject deletion path covering trace and eval stores
A deletion request must reach the trace store and the eval set. It must also reach the primary index.
Subprocessor visibility on the model and embedding providers
The vendor risk reviewer needs the full chain. The provider named on the contract is only the first link.
Which patterns and failure modes tie in?#
How long does a first version take?#
4-8 weeks to a pilot that answers real questions against a live corpus. Add 2-4 weeks to build change-data-capture ingest and a real eval set. That work separates a pilot from a one-time batch load behind a demo script.
When is this the wrong shape?#
- Your corpus is small and stable enough for a person to read. Or a decent keyword search bar already finds the answer.
- You need the system to take actions on your behalf, beyond answering questions. That need calls for the agentic workflow runner.
- Nobody on your side will keep the eval set current after launch. Without a maintained eval set, a grounded-answering system degrades. Nobody notices until a customer does.
Sources
- arXiv - Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
The paper that named retrieval-augmented generation as a technique. Retrieved 2026-08-24.
- arXiv - Passage Re-ranking with BERT
The paper that established cross-encoder scoring as the second-stage reranking method our retrieve-then-rerank pattern applies. Retrieved 2026-09-17.
A Ship Audit checks which of these an existing or planned system needs. It ranks them by what is most likely to break first.