From Notebook to Production: An Extraction Model You Could Trust

The problem

A mid-market healthtech had built a model to pull structured fields out of clinical intake documents. Patient demographics, referring provider, diagnosis codes, insurance details. The source was whatever a referring office sent: PDFs, faxes, scanned referral forms. In a data scientist’s notebook, against a clean sample of well-formatted documents, it looked ready to ship. Extraction accuracy on that sample was high enough that clinical operations was already planning to retire the manual re-keying process it would replace.

Real intake traffic didn’t look like the notebook sample. Faxes came in skewed and low-resolution. Referral forms varied by sending office, and some used field labels the model had never seen. On that long tail, the model didn’t fail loudly. It filled in a plausible-looking value where the real one was unclear or missing. A wrong diagnosis code or a swapped date of birth doesn’t throw an error. It silently enters the clinical record looking exactly as confident as a correct one. Nobody had measured how often this happened, because there was no eval harness. There was only the one notebook sample everyone had already looked at. The documents also carried protected health information, so the team could not route them through an external extraction API to buy time. Whatever they built had to keep PHI inside their own infrastructure.

What we ruled out, and what we built

The fastest option was wrapping the existing notebook code in an API and shipping it as is. We ruled it out. There was no eval harness, so the hallucinationA hallucination is a confident, fluent output that is factually wrong or unsupported. rate on real traffic was unmeasured and unmanaged. With no regression safety net, a future model or prompt change could worsen the long tail with nobody noticing until a clinician did. The other option was a rules-based extractor: deterministic, easy to audit, and appealing to a compliance-minded team. We ruled that out too. The fax-format long tail was too varied for hand-written rules to cover. Each new referring office’s layout would demand another round of rule-writing that never quite caught up.

We built a production extraction pipeline around three pieces. First, an eval harness built from a labeled sample of real intake documents, not the clean notebook sample. Faxes, scans, and edge-case layouts pulled from actual traffic. Clinical operations hand-labeled each one with the correct field values. This became the yardstick for every later change. A prompt tweak, a model swap, a preprocessing fix: all measured against the same held-out set before it touched production. Second, confidence gating on every extracted field. When the model’s confidence on a field fell below a calibrated threshold, that field went to a human reviewer. The whole document did not. Third, we kept every document and every model call inside the healthtech’s own VPC. No PHI left their infrastructure to reach an external API. The extraction model ran on infrastructure they controlled, which mattered to their security team as much as the accuracy numbers did.

The eval harness ran in CI on every change, the same way a test suite gates a code change. That turned “is this update better or worse” from a question someone eyeballed after deployment into a number the team saw before merging.

The outcome

Field-level hallucination rate on the held-out real-document sample dropped from 12% to 3%. The gain came less from a better underlying model than from the ability to see where it was wrong and gate on that. The confidence threshold caught the fields the model was unsure about. The eval harness caught the changes that would have made things worse before they shipped. The system went from notebook to production traffic in eight weeks. Only low-confidence fields went to a human, so the manual review queue shrank 60% against the fully manual re-keying process it replaced. A human still reviewed exactly the cases where the model’s own confidence said one was needed. Clinical operations put the saved re-keying work at roughly 1,200 engineer-hours a year, freed up for other intake work.

For the CFO, the math is straightforward. The engineer-hours saved against the manual re-keying baseline, set against the eight-week build cost, put payback inside a single quarter. It’s a number built from a documented, repeatable eval process, not a black-box accuracy claim. The finance team could walk through it with clinical operations. For IT and security, PHI never leaving the VPC mattered as much as the accuracy gain. The pipeline slotted into the existing EHR intake flow with no new external data-sharing agreement and no new HIPAA risk to assess. No new external party touched the data. For operations, confidence gating plus an eval suiteAn eval suite is a versioned set of test cases that measures whether outputs are good. in CI means degradation is graceful, not silent. A bad model update gets caught by the eval harness before it reaches production. The residual 3% of uncertain extractions get caught by a human reviewer before they enter the clinical record. Nobody has to catch a wrong code downstream.

The Head of Clinical Operations put it plainly: the win wasn’t a smarter model. For the first time, the team could measure when the model was wrong, at the field level, on real documents. That’s what turned a notebook demo into something the clinical intake process could depend on.