From Notebook to Production: An Extraction Model You Could Trust

The problem

A mid-market healthtech had built a model that pulled structured fields (patient demographics, referring provider, diagnosis codes, insurance details) out of clinical intake documents: PDFs, faxes, scanned referral forms, whatever a referring office happened to send. In a data scientist’s notebook, against a clean sample of well-formatted documents, it looked ready to ship. Extraction accuracy on that sample was high enough that the clinical operations team was already planning to retire the manual re-keying process it would replace.

Real intake traffic didn’t look like the notebook sample. Faxes came in skewed and low-resolution. Referral forms varied by sending office, some using field labels the model had never seen. On that long tail, the model didn’t fail loudly. It filled in a plausible-looking value where the real one was unclear or missing. A wrong diagnosis code or a swapped date of birth doesn’t throw an error; it silently enters the clinical record looking exactly as confident as a correct one. Nobody had measured how often this was happening, because there was no eval harness, just the one notebook sample everyone had already looked at. And because the documents contained protected health information, the team couldn’t just route them through an external extraction API to buy time; whatever they built had to keep PHI inside their own infrastructure.

What we ruled out, and what we built

The fastest option on the table was wrapping the existing notebook code in an API and shipping it as-is. We ruled it out: there was no eval harness, so the hallucination rate on real traffic was unmeasured and unmanaged, and no regression safety net meant every future model or prompt change could quietly make the long tail worse with nobody noticing until a clinician did. The other option on the table was a rules-based extractor: deterministic, easy to audit, and appealing to a compliance-minded team. We ruled that out too. The fax-format long tail was too varied for hand-written rules to cover, and every new referring office’s document layout would mean another round of rule-writing that never quite catches up.

We built a production extraction pipeline around three pieces. First, an eval harness built from a labeled sample of real intake documents, not the clean notebook sample, but faxes, scans, and edge-case layouts pulled from actual traffic, each hand-labeled with the correct field values by the clinical operations team. This became the yardstick for every subsequent change: a prompt tweak, a model swap, a preprocessing fix, all measured against the same held-out set before it touched production. Second, confidence gating on every extracted field: when the model’s confidence on a field fell below a calibrated threshold, that field — not the whole document — was routed to a human reviewer instead of being written straight into the record. Third, we kept every document and every model call inside the healthtech’s own VPC. No PHI left their infrastructure to reach an external API; the extraction model ran on infrastructure they controlled, which mattered as much to their security team as the accuracy numbers did.

The eval harness ran in CI on every change, the same way a test suite gates a code change. That turned “does this update make extraction better or worse” from a question someone eyeballed after deployment into a number the team saw before merging.

The outcome

Field-level hallucination rate on the held-out real-document sample dropped from 12% to 3%, driven less by a better underlying model than by the ability to see where it was wrong and gate on that. The confidence threshold caught the fields the model was actually unsure about, and the eval harness caught the changes that would have made things worse before they shipped. The system went from notebook to production traffic in eight weeks. Because low-confidence fields were the only ones routed to a human, the manual review queue shrank 60% compared to the fully-manual re-keying process it replaced, while still keeping a human in the loop exactly where the model’s confidence said one was needed. Clinical operations estimated the reduction in re-keying work at roughly 1,200 engineer-hours a year freed up for other intake work.

For the CFO, the math is straightforward: the engineer-hours saved against the manual re-keying baseline, set against the eight-week build cost, put payback inside a single quarter. Unlike a black-box accuracy claim, it’s a number built from a documented, repeatable eval process the finance team could actually walk through with clinical operations. For IT and security, the fact that PHI never left their VPC mattered as much as the accuracy improvement; the pipeline slotted into their existing EHR intake flow without opening a new external data-sharing agreement or a new HIPAA risk to assess, because there was no new external party in the data path at all. For operations, the combination of confidence gating and an eval suite running in CI means degradation is graceful, not silent: a bad model update gets caught by the eval harness before it reaches production, and the residual 3% of genuinely uncertain extractions get caught by a human reviewer before they enter the clinical record, rather than after a clinician catches a wrong diagnosis code downstream.

The Head of Clinical Operations put it plainly: the win wasn’t a smarter model. It was that for the first time, the team could actually measure when the model was wrong, at the field level, on real documents. That’s what turned a notebook demo into something the clinical intake process could depend on.