Document extraction with a human review loop
Turn inbound documents into structured records. A person reviews only the cases where the model is unsure.
“We get PDFs, forms, or emails. Someone keys the data into our system by hand. Can software read them for us?”
What does the system look like?#
How does a request move through it?#
- Intake and normalize
Documents arrive by upload, email, or scan. Preprocessing and OCR turn them into a consistent text form before extraction runs.
- Extract to a fixed schema
The model fills a defined structured-output schema, field by field. It does not return free text for a second process to parse.
- Validate the contract
The validator checks every field against its schema, type, and required status before anything downstream trusts it. A validation failure is a distinct, counted event.
- Route by confidence
A confidence score per field or per document decides the path. The extraction ships to the system of record or waits on a human.
- Human review on the uncertain tail
A reviewer sees only the cases the model flagged. The reviewer corrects or approves each one. The correction becomes the ground truth for the drift monitor.
- Write and monitor drift
The approved record writes to the system of record. The team tracks the gap between the model's draft and the accepted record over time. The gap is never assumed to be zero.
What are the pieces, and what breaks without each one?#
Preprocessing & OCR
Normalize every inbound format (PDF, scan, email) into a consistent text form before extraction runs.
Breaks without it: The extractor sees a different input shape per document type. Its accuracy then depends on format. Nobody measures that effect on its own.
Structured extractor
Fill a fixed schema field by field from the document. Do not return free text for a second pass to parse.
Breaks without it: A response in the wrong shape breaks the next step in the chain. The extracted content may still be right.
Schema validator
Check every extracted field against its type and required-ness before anything downstream reads it.
Breaks without it: A malformed or missing field reaches the system of record without a warning. It does not fail loudly enough for anyone to fix it.
Confidence router
Send the uncertain tail to a human. Let a well-calibrated high-confidence path ship straight through.
Breaks without it: Either every document needs a human, which is the manual process this pipeline replaces, or none do. Then wrong data ships with confidence.
Human review queue
Correct or approve only the cases the model flagged. Capture each correction as ground truth.
Breaks without it: Nobody checks a model regression on the low-confidence tail. It ships.
Drift monitor
Measure the edit distance between the model's draft and the record a reviewer accepted. Track it over time.
Breaks without it: Extraction quality declines after a document-format change or a model update. Nobody notices until a downstream report is wrong.
Which choices need a decision?#
| Decision | Default choice | Why |
|---|---|---|
| OCR / preprocessing | A managed OCR API to start. Self-host only when volume or data-residency requirements justify it | OCR quality differences matter far less than a consistent, well-tested normalization step. Put that step in place early. |
| Extraction model | A prompted frontier model against a strict schema. Use a fine-tuned model only when volume and stability justify the retraining cost | Fine-tuning pays off once the document shape is stable and volume is high. Before that, it is an ongoing retraining burden for a marginal accuracy gain. |
| Review UI | Route the low-confidence queue into the tool the team already uses for case work. Skip the new bespoke reviewer screen | A reviewer who must learn a second tool reviews less carefully and less often. A reviewer in the existing workflow does better. |
| System-of-record write | Write through a staged queue with its own validation. Skip the direct write from the extractor | A staged write gives you a place to catch a validation failure. Bad data then never lands in a system other teams depend on. |
What drives the bill?#
- Extraction model calls per document
- Reviewer minutes per low-confidence document
- Reprocessing on schema-validation failures
Dominant cost Reviewer labor, once the confidence router is tuned well. The extraction call is usually the cheap part of a well-calibrated pipeline.
The lever Raise the confidence floor with tighter schemas and better preprocessing to cut review volume. Do not lower the bar for high confidence.
Model your own numbers with the AI Cost Calculator →How do you know it works?#
Confirms every extracted field is well-formed before it reaches the system of record.
Edge-case coverage in the labelled set
Tracks whether the eval set is hard. An eval set of easy documents proves little.
Edit distance on accepted output
Measures how much a reviewer had to change before accepting the model's draft.
Blocks a prompt or schema change that regresses a known document type.
What will your reviewers ask about this?#
Strip or redact fields that the extraction schema does not need. Do this before the document reaches the model.
Logging & retention with redaction
Extracted documents often carry more sensitive fields than the feature needs. The team states retention and redaction rules in writing.
Named accountable owner for extraction errors
A wrong field in the system of record has a specific owner and a monitoring signal. "The model decided" is not an owner.
Which patterns and failure modes tie in?#
How long does a first version take?#
4-6 weeks to a pipeline that extracts and routes by confidence. The review queue and drift monitor usually take another 2-4 weeks to tune against real documents.
When is this the wrong shape?#
- Your documents are already structured, like a form with fixed fields at fixed positions. A deterministic parser costs less and never hallucinates a field.
- Nobody will staff the human review queue for the low-confidence tail. An unstaffed queue becomes a backlog that never shrinks.
- A wrong extracted value costs nothing, because nothing downstream depends on it. A spot-check is enough for that case. You do not need this pipeline.
Sources
- JSON Schema - JSON Schema, Draft 2020-12
The schema specification a tool description must satisfy under our rule bank. Retrieved 2026-08-24.
- Stanford CRFM - Holistic Evaluation of Language Models (HELM)
The benchmark suite our check bank draws its evaluation method from. Retrieved 2026-08-24.
A Ship Audit checks which of these an existing or planned system needs. It ranks them by what is most likely to break first.