The Eval Stack
How to know an AI system works before it ships: the eval layers, plus the ways an LLM judge can quietly lie to you.
21 min read Read → Topics
Evals & Observability
Shipping an AI feature without an eval suiteAn eval suite is a versioned set of test cases that measures whether outputs are good. means every prompt or model change is a guess about whether quality went up or down. This is our work on evals and the observabilityObservability captures traces of every prompt, retrieval, tool call, and response for debugging. that catches regressions after ship. Both belong in CI.
Start with The Eval Stack if you ship AI features. It's for engineers who need to know whether a prompt or model change made quality better or worse.
Guides#
Comparisons#
Agents vs Pipelines: When Autonomy Is Worth the Reliability Cost
Agentic autonomy and deterministic pipelines aren't a spectrum of ambition. They trade flexibility against reliability, and here's which your workflow needs.
7 min read Read →Insights#
What an AI Feature Actually Costs in Production
Token costs that look trivial in a demo compound fast at scale. Here's how to make cost a first-class metric instead of a surprise on the invoice.
8 min read Read →Evals Before You Ship: Why AI Features Need Tests Too
Shipping an AI feature without an eval suite in CI means every prompt tweak is a guess. Here's what an eval suite actually needs to cover.
7 min read Read →Case studies#
From Notebook to Production: An Extraction Model You Could Trust
A healthtech's extraction model dazzled in a notebook, then hallucinated on real traffic. An eval harness cut hallucinationsA hallucination is a confident, fluent output that is factually wrong or unsupported. from 12% to 3% in 8 weeks.
7 min read Read →Patterns#
Golden-set gate in CI
A fixed, human-labelled set of real cases runs automatically in CI on every prompt or model change. Each case carries a specific, checkable expected property, not a vibe. A drop against the current pass-rate threshold blocks the merge the same way a failing unit test would. A regression gets caught before release, not after a user complaint.
Read →Trace-first observability
One trace ID follows a single request across every hop it takes: retrieval, every model call, every tool call. It's logged with enough detail to reconstruct exactly what happened after the fact. A sampled slice of traces, weighted toward low-confidence or error-flagged runs, lands in a queue a human reviews on a regular schedule. That happens well before any complaint forces someone to go looking.
Read →Failure modes#
Vibes-based prompt regression
Without a labelled eval set, the change was graded against whatever two or three examples the author happened to have open. That is not a test. It's an anecdote. A regression anywhere outside that narrow, unrepresentative sample ships straight to production undetected.
Read →Judge prefers its own output
A judge from the same model family as the generator tends to rate that family's output favorably. This is self-preference bias. A single vague rubric ('is this good?') collapses almost everything to a passing score, so the eval suite stops being able to tell a real regression from noise.
Read →