# Evals Before You Ship: Why AI Features Need Tests Too Source: https://customlabs.io/insights/evals-before-you-ship/ Updated: 2026-09-11 Insights June 23, 2026 · 7 min read · By CustomLabs Engineering · Updated September 11, 2026 · # Evals Before You Ship: Why AI Features Need Tests Too [evals](https://customlabs.io/topics/evals-observability/)[observability](https://customlabs.io/topics/evals-observability/)[ci](https://customlabs.io/topics/evals-observability/) Key takeaways - → Without an [eval suite](https://customlabs.io/glossary/eval-suite/) in CI, every prompt tweak, model swap, or provider update is an untested bet - → Eval cases need to cover the precise happy path, past incidents, adversarial/out-of-scope inputs, and cost and latency - → Judge non-deterministic output with a mix of structural checks, rule-based checks, and [LLM-as-judge](https://customlabs.io/glossary/llm-as-judge/) for open-ended quality - → A suite that's 100% LLM-as-judge is expensive and noisy; lean on structural and rule-based checks first - → The suite must run automatically on every change that could affect output, and fail the build on regression An AI feature without an eval suite in CI means every prompt tweak, model swap, or provider update is an untested bet — and you find out you lost it from a support ticket. AI features are software and deserve the same discipline: a fixed set of cases with known-good expectations that fails the build on regression, covering the happy path, past incidents, adversarial inputs, and cost and latency, not just whether the output looks plausible. Traditional software has a straightforward answer for “did this change break anything”: tests, run in CI, on every commit. AI features get the same discipline here, not a lower bar just because the output is harder to grade than an exact-match assertion. That’s exactly what the eval suite below needs to cover. ## What do eval cases actually need to cover? **The happy path, precisely.** Start with the inputs the feature is built for, and pin down what a correct response looks like for each, not “sounds plausible,” but a specific, checkable property: the right document is retrieved, the requested field is extracted correctly, the tool call has the right arguments. Vague success criteria produce eval suites that pass even when quality has quietly degraded. **The edge cases that already bit you.** Every AI feature we’ve maintained has a running list of “the time it did something weird” incidents: a malformed input, an ambiguous request, a document with a table that broke extraction. Every one of those belongs in the eval suite as a regression test, the same way you’d add a unit test for a bug you just fixed. Otherwise you fix the same failure mode twice. **Adversarial and out-of-scope inputs.** What should the feature do when asked something it’s not built for, or something it shouldn’t answer, like a [prompt injection](https://customlabs.io/glossary/prompt-injection/) attempt, a request for data the user shouldn’t see, or a question entirely outside the feature’s domain? “Doesn’t crash” is the floor; “declines gracefully and doesn’t leak anything” is the actual bar. These cases rarely show up in a demo because nobody adversarially tests their own demo, which is exactly why they need to be in the suite deliberately. Cost and latency matter alongside correctness. A prompt change that improves accuracy by making the model reason at ten times the length is a real tradeoff, not a pure win, and an eval suite that only checks correctness will never surface that. Track token cost and response time per case alongside the pass/fail, so a “successful” change that quietly triples your inference bill gets caught before it ships, not after the invoice arrives. ## How do you judge non-deterministic AI output? The hard part of AI evals, compared to traditional tests, is that outputs aren’t exact-string-match deterministic: the same prompt can produce two differently-worded but equally correct answers. A few approaches that actually work in practice, usually combined: - **Structural checks** wherever the output has structure: the right fields are present, a tool call has valid arguments, a classification lands in the right bucket. This is the cheapest, most reliable check and should cover as much of the suite as possible. - **Rule-based checks** for properties you can assert directly, like response length within bounds, required disclaimers present, or no PII in the output. - **LLM-as-judge** for genuinely open-ended quality questions, with a judge prompt that’s itself versioned, reviewed, and periodically checked against human judgment, because a judge that’s silently drifted is worse than no judge, since it gives false confidence. The mix matters more than any single technique. A suite that’s 100% LLM-as-judge is expensive and noisy; a suite that’s 100% exact-match will reject correct answers that are phrased differently and train the team to ignore its failures. ## How do you wire an eval suite into CI? An eval suite that only runs when someone remembers to run it manually will stop running within a month. The suite needs to run automatically on every change that could plausibly affect output (prompt edits, model version bumps, retrieval or tool changes), and the build needs to fail, or at minimum flag a human, when the pass rate or cost/latency profile moves in the wrong direction. That’s the same bar we hold traditional code to, and there’s no good reason to hold AI code to a lower one just because “testing AI is different.” It’s different in technique, not in whether it should happen before you ship. This is table stakes in every custom AI feature we build now: the eval suite ships alongside the feature, wired into CI, so “did this get better or worse” has a concrete answer on every pull request, not a vibe check after it’s already live. Two specific ways an eval suite goes wrong even when it exists: see the field guide entries on [vibes-based prompt regression](https://customlabs.io/failure-modes/vibes-based-prompt-regression/) and [judge prefers its own output](https://customlabs.io/failure-modes/judge-prefers-its-own-output/). Questions ## FAQ Answers to the questions this piece raises. 01 Why do AI features need an eval suite instead of just testing manually? + Because AI features keep changing (through prompt tweaks, model swaps, or provider updates), and an eval suite that runs in CI catches regressions automatically, while manual spot-checks only tell you the demo looked right once. 02 What should an eval suite actually cover? + The precise happy path, the edge cases that already caused incidents, adversarial and out-of-scope inputs, and cost and latency alongside correctness. 03 How do you judge AI output that isn't exact-match deterministic? + By combining structural checks, rule-based checks, and LLM-as-judge for open-ended quality, leaning on structural and rule-based checks as much as possible, since a suite that's 100% LLM-as-judge is expensive and noisy. Related services [Custom Development](https://customlabs.io/services/custom-development/)[AI Integrations](https://customlabs.io/services/ai-integrations/) Related tools [AI Readiness Scorecard](https://customlabs.io/tools/ai-readiness/) A prompt tweak without a test is a guess. Check whether your current suite covers the happy path and past incidents. The Eval Stack lays out what a suite needs and how to wire it into CI. [Run the AI Readiness Scorecard →](https://customlabs.io/tools/ai-readiness/) [Read the Eval Stack →](https://customlabs.io/evals/) Written by [CustomLabs Engineering](https://customlabs.io) Applied-AI engineering team CustomLabs is a small, senior-only studio that embeds with client teams and ships eval-tested, model-agnostic AI systems into production in weeks, not quarters. Every insight reflects work and lessons from the studio's own engagements — the people who write the code write the words. Our products [CodeHerder](https://codeherder.com)[CostMon](https://costmon.com)[FreeTier](https://freetier.co)[GreatAPIs](https://greatapis.com)[Beemy](https://beemy.co)[CustomHosted](https://customhosted.com) Read next [June 30, 2026 · 8 min read ### What an AI Feature Actually Costs in Production Token costs that look trivial in a demo compound fast at scale. Here's how to make cost a first-class metric instead of a surprise on the invoice. costproductionobservability Read →](https://customlabs.io/insights/what-ai-actually-costs/)[August 12, 2026 · 9 min read ### What Actually Moved in the EU AI Act's 2026 Deadline Change Regulation (EU) 2026/1744 pushed the AI Act's high-risk deadline to December 2027. What moved, what didn't, and why the old schedule still pays off. governancecomplianceeu-ai-act Read →](https://customlabs.io/insights/ai-act-high-risk-deadline-moved/)[July 24, 2026 · 10 min read ### Prompt Injection Is a Data Problem: A Threat Model You Can Ship Against Prompt injection can't be filtered away: the model can't reliably tell instructions from data. Here's the actual threat model and the controls that hold up. securityprompt-injectionproduction Read →](https://customlabs.io/insights/prompt-injection-threat-model/)