Document extraction with a human review loop
Turn inbound documents into structured records, with a person reviewing only the cases the model is actually unsure about.
“We get PDFs, forms, or emails and someone manually keys the data into our system. Can this just get read automatically?”
What does the system look like?#
How does a request move through it?#
- Intake and normalize
Documents arrive by upload, email, or scan and are preprocessed and OCR'd into a consistent text representation before extraction ever runs.
- Extract to a fixed schema
The model fills a defined structured-output schema, field by field, rather than returning free text a second process has to parse and hope.
- Validate the contract
Every field is checked against its schema, type, and required-ness before anything downstream trusts it; a validation failure is a distinct, counted event, not a silent pass-through.
- Route by confidence
A per-field or per-document confidence score decides whether the extraction ships straight to the system of record or waits on a human.
- Human review on the uncertain tail
A reviewer sees only the cases the model flagged, corrects or approves them, and that correction becomes the ground truth for the drift monitor.
- Write and monitor drift
The approved record writes to the system of record, and the gap between the model's draft and what the reviewer actually accepted is tracked over time, not assumed to be zero.
What are the pieces, and what breaks without each one?#
Preprocessing & OCR
Normalize every inbound format (PDF, scan, email) into a consistent text representation before extraction runs.
Breaks without it: The extractor sees a different input shape per document type and its accuracy becomes format-dependent in ways nobody measures separately.
Structured extractor
Fill a fixed schema field by field from the document, instead of returning free text for a second pass to parse.
Breaks without it: A wrongly-shaped response breaks the next step in the chain regardless of whether the extracted content itself was right.
Schema validator
Check every extracted field against its type and required-ness before anything downstream reads it.
Breaks without it: A malformed or missing field reaches the system of record silently instead of failing loud enough for someone to fix it.
Confidence router
Send the uncertain tail to a human and let a well-calibrated high-confidence path ship straight through.
Breaks without it: Every document needs a human, which is the manual process this exists to replace, or none do, which ships wrong data with confidence.
Human review queue
Correct or approve only the cases the model flagged, with the correction captured as ground truth.
Breaks without it: A model regression on the low-confidence tail has nobody actually checking it, and it ships anyway.
Drift monitor
Measure the edit distance between the model's draft and what a reviewer actually accepted, tracked over time.
Breaks without it: Extraction quality quietly declines after a document-format change or a model update and nobody notices until a downstream report is wrong.
Where does this need a decision, not a default?#
| Decision | Default choice | Why |
|---|---|---|
| OCR / preprocessing | A managed OCR API to start, self-hosted only once volume or data-residency requirements justify it | OCR quality differences matter far less than getting a consistent, well-tested normalization step in place quickly. |
| Extraction model | A prompted frontier model against a strict schema, not a fine-tuned model, until volume and stability justify the retraining cost | Fine-tuning pays off once the document shape is stable and volume is high; before that, it is an ongoing retraining burden for a marginal accuracy gain. |
| Review UI | Route the low-confidence queue into whatever tool the team already uses for case work, not a new bespoke reviewer screen | A reviewer who has to learn a second tool reviews less carefully and less often than one working inside their existing workflow. |
| System-of-record write | Write through a staged queue with its own validation, not a direct write from the extractor | A staged write gives you a place to catch a validation failure before it lands as bad data in a system other teams depend on. |
What actually drives the bill?#
- Extraction model calls per document
- Reviewer minutes per low-confidence document
- Reprocessing on schema-validation failures
Dominant cost Reviewer labor once the confidence router is tuned well, because the extraction call itself is usually the cheap part of a well-calibrated pipeline.
The lever Raise the confidence floor with tighter schemas and better preprocessing to cut review volume, not by lowering the bar for what counts as high confidence.
Model your own numbers with the AI Cost Calculator →How do you know it actually works?#
Confirms every extracted field is well-formed before it reaches the system of record.
Edge-case coverage in the labelled set
Tracks whether the eval set is actually hard, not quietly all easy documents.
Edit distance on accepted output
Measures how much a reviewer actually had to change before accepting the model's draft.
Blocks a prompt or schema change that regresses a known document type.
What will your reviewers ask about this?#
Strip or redact fields the extraction schema doesn't actually need before the document reaches the model.
Logging & retention with redaction
Extracted documents often carry more sensitive fields than the feature needs; retention and redaction are stated, not assumed.
Named accountable owner for extraction errors
A wrong field that reaches the system of record has a specific owner and a monitoring signal, beyond "the model decided".
Which patterns and failure modes tie in?#
How long does a first version actually take?#
4-6 weeks to a pipeline that extracts and routes by confidence; the review queue and drift monitor typically take another 2-4 weeks to tune against real documents.
When is this the wrong shape?#
- Your documents are already structured, like a form with fixed fields at fixed positions. A deterministic parser is cheaper and never hallucinates a field.
- Nobody will staff the human review queue for the low-confidence tail. An unstaffed queue just becomes a backlog that never gets shorter.
- A wrong extracted value costs nothing when it is wrong, because nothing downstream actually depends on it being correct. A spot-check is enough for that case, not this whole pipeline.
Sources
- JSON Schema - JSON Schema, Draft 2020-12
The schema specification a tool description must satisfy under our rule bank. Retrieved 2026-08-24.
- Stanford CRFM - Holistic Evaluation of Language Models (HELM)
The benchmark suite our check bank draws its evaluation method from. Retrieved 2026-08-24.
A Ship Audit checks which of these an existing or planned system actually needs, against what is most likely to break first.