What an AI Feature Actually Costs in Production
Token costs that look trivial in a demo compound fast at scale. Here's how to make cost a first-class metric instead of a surprise on the invoice.
Read → Topics
Evals & Observability
Shipping an AI feature without an eval suite means every prompt or model change is a guess about whether quality went up or down. This is our work on evals, CI, and the observability that catches regressions after ship.
Insights
Evals Before You Ship: Why AI Features Need Tests Too
Shipping an AI feature without an eval suite in CI means every prompt tweak is a guess. Here's what an eval suite actually needs to cover.
Read →Case studies
From Notebook to Production: An Extraction Model You Could Trust
A mid-market healthtech had a document-extraction model that dazzled in a notebook and hallucinated on real intake traffic. An eval harness and confidence gating took hallucinations from 12% to 3% and shipped it in 8 weeks.
Read →Failure modes
Vibes-based prompt regression
Without a labelled eval set, the change was graded against whatever two or three examples the author happened to have open — which is not a test, it's an anecdote. A regression anywhere outside that narrow, unrepresentative sample ships straight to production undetected.
Read →Judge prefers its own output
A judge from the same model family as the generator tends to rate that family's output favorably — self-preference bias — and a single vague rubric ('is this good?') collapses almost everything to a passing score, so the eval suite stops being able to tell a real regression from noise.
Read →Patterns
Golden-set gate in CI
A fixed, human-labelled set of real cases — each with a specific, checkable expected property, not a vibe — runs automatically in CI on every prompt or model change. A drop against the current pass-rate threshold blocks the merge the same way a failing unit test would, so a regression is caught before release, not after a user complaint.
Read →Trace-first observability
One trace ID follows a single request across every hop it takes — retrieval, every model call, every tool call — logged with enough detail to reconstruct exactly what happened after the fact. A sampled slice of traces, weighted toward low-confidence or error-flagged runs, lands in a queue a human actually reviews on a schedule, not only when a complaint forces someone to go looking.
Read →