The Eval Stack
How to know an AI system works before it ships. Read the eval layers, the checks behind them, and the ways an LLM judge lies.
21 min read Read → Topics
Evals & Observability
Shipping an AI feature without an eval suiteAn eval suite is a versioned set of test cases that measures whether outputs are good. means every prompt or model change is a guess about whether quality went up or down. This is our work on evals and the observabilityObservability captures traces of every prompt, retrieval, tool call, and response for debugging. that catches regressions after ship. Both belong in CI.
Start with The Eval Stack if you ship AI features. It's for engineers who need to know whether a prompt or model change made quality better or worse.
Guides#
Comparisons#
Agents vs Pipelines: When Autonomy Is Worth the Reliability Cost
Agentic autonomy and deterministic pipelines aren't a spectrum of ambition. They trade flexibility against reliability, and your workflow decides which.
7 min read Read →Insights#
What Our Coding Agents Did to Get Past the Build Checks
A coding agent optimises for the check and skips the goal. Real incidents from our own delivery record show how a green check hid a wrong page.
9 min read Read →What an AI Feature Costs in Production
Token costs that look trivial in a demo compound fast at scale. Make cost a first-class metric instead of a surprise on the invoice.
7 min read Read →Evals Before You Ship: Why AI Features Need Tests Too
Shipping an AI feature without an eval suite in CI means every prompt tweak is a guess. This piece lists what an eval suite needs to cover.
7 min read Read →Case studies#
From Notebook to Production: An Extraction Model You Could Trust
A healthtech's extraction model dazzled in a notebook, then hallucinated on real traffic. An eval harness cut hallucinationsA hallucination is a confident, fluent output that is factually wrong or unsupported. from 12% to 3% in 8 weeks.
6 min read Read →Patterns#
Golden-set gate in CI
A fixed set of human-labelled real cases runs in CI on every prompt or model change. Each case has a specific, checkable expected property. A drop below the pass-rate threshold blocks the merge, as a failing unit test would. The team catches a regression before release and avoids a user complaint.
Read →Trace-first observability
One trace ID follows a single request across every hop, from retrieval to each model call and tool call. The system logs enough detail to rebuild what happened. A sampled slice of traces goes to a human review queue on a regular schedule. The sample favors low-confidence or error-flagged runs. Review starts long before a complaint forces a search.
Read →Failure modes#
Vibes-based prompt regression
Without a labelled eval set, the change was graded against whatever two or three examples the author happened to have open. That is not a test. It's an anecdote. A regression anywhere outside that narrow, unrepresentative sample ships straight to production undetected.
Read →Judge prefers its own output
A judge from the same model family as the generator tends to rate that family's output favorably. This is self-preference bias. A single vague rubric ('is this good?') collapses almost everything to a passing score. The eval suite then cannot tell a real regression from noise.
Read →