# Evals & Observability Source: https://customlabs.io/topics/evals-observability/ Updated: 2026-09-14 Topics # Evals & Observability Shipping an AI feature without an [eval suite](https://customlabs.io/glossary/eval-suite/) means every prompt or model change is a guess about whether quality went up or down. This is our work on evals and the [observability](https://customlabs.io/glossary/observability/) that catches regressions after ship. Both belong in CI. Start with The Eval Stack if you ship AI features. It's for engineers who need to know whether a prompt or model change made quality better or worse. In the Handbook [01 Decide](https://customlabs.io/handbook/decide/)[02 Design](https://customlabs.io/handbook/design/)[03 Build](https://customlabs.io/handbook/build/)[04 Evaluate](https://customlabs.io/handbook/evaluate/)[05 Operate](https://customlabs.io/handbook/operate/) ## Guides ### The Eval Stack How to know an AI system works before it ships: the eval layers, plus the ways an LLM judge can quietly lie to you. 21 min read [Read →](https://customlabs.io/evals/) ## Comparisons July 22, 2026 ### Agents vs Pipelines: When Autonomy Is Worth the Reliability Cost Agentic autonomy and deterministic pipelines aren't a spectrum of ambition. They trade flexibility against reliability, and here's which your workflow needs. 7 min read [Read →](https://customlabs.io/compare/agents-vs-pipelines/) ## Insights June 30, 2026 ### What an AI Feature Actually Costs in Production Token costs that look trivial in a demo compound fast at scale. Here's how to make cost a first-class metric instead of a surprise on the invoice. 8 min read [Read →](https://customlabs.io/insights/what-ai-actually-costs/) June 23, 2026 ### Evals Before You Ship: Why AI Features Need Tests Too Shipping an AI feature without an eval suite in CI means every prompt tweak is a guess. Here's what an eval suite actually needs to cover. 7 min read [Read →](https://customlabs.io/insights/evals-before-you-ship/) ## Case studies June 30, 2026 ### From Notebook to Production: An Extraction Model You Could Trust A healthtech's extraction model dazzled in a notebook, then hallucinated on real traffic. An eval harness cut [hallucinations](https://customlabs.io/glossary/hallucination/) from 12% to 3% in 8 weeks. 7 min read [Read →](https://customlabs.io/case-studies/extraction-notebook-to-production/) ## Patterns ### Golden-set gate in CI A fixed, human-labelled set of real cases runs automatically in CI on every prompt or model change. Each case carries a specific, checkable expected property, not a vibe. A drop against the current pass-rate threshold blocks the merge the same way a failing unit test would. A regression gets caught before release, not after a user complaint. [Read →](https://customlabs.io/patterns/golden-set-gate-in-ci/) ### Trace-first observability One trace ID follows a single request across every hop it takes: retrieval, every model call, every tool call. It's logged with enough detail to reconstruct exactly what happened after the fact. A sampled slice of traces, weighted toward low-confidence or error-flagged runs, lands in a queue a human reviews on a regular schedule. That happens well before any complaint forces someone to go looking. [Read →](https://customlabs.io/patterns/trace-first-observability/) ## Failure modes ### Vibes-based prompt regression Without a labelled eval set, the change was graded against whatever two or three examples the author happened to have open. That is not a test. It's an anecdote. A regression anywhere outside that narrow, unrepresentative sample ships straight to production undetected. [Read →](https://customlabs.io/failure-modes/vibes-based-prompt-regression/) ### Judge prefers its own output A judge from the same model family as the generator tends to rate that family's output favorably. This is self-preference bias. A single vague rubric ('is this good?') collapses almost everything to a passing score, so the eval suite stops being able to tell a real regression from noise. [Read →](https://customlabs.io/failure-modes/judge-prefers-its-own-output/)