Evals Before You Ship: Why AI Features Need Tests Too

An AI feature without an eval suite in CI means every prompt tweak, model swap, or provider update is an untested bet — and you find out you lost it from a support ticket. AI features are software and deserve the same discipline: a fixed set of cases with known-good expectations that fails the build on regression, covering the happy path, past incidents, adversarial inputs, and cost and latency, not just whether the output looks plausible.

Traditional software has a straightforward answer for “did this change break anything”: tests, run in CI, on every commit. AI features get the same discipline here, not a lower bar just because the output is harder to grade than an exact-match assertion. That’s exactly what the eval suite below needs to cover.

What do eval cases actually need to cover?

The happy path, precisely. Start with the inputs the feature is built for, and pin down what a correct response looks like for each, not “sounds plausible,” but a specific, checkable property: the right document is retrieved, the requested field is extracted correctly, the tool call has the right arguments. Vague success criteria produce eval suites that pass even when quality has quietly degraded.

The edge cases that already bit you. Every AI feature we’ve maintained has a running list of “the time it did something weird” incidents: a malformed input, an ambiguous request, a document with a table that broke extraction. Every one of those belongs in the eval suite as a regression test, the same way you’d add a unit test for a bug you just fixed. Otherwise you fix the same failure mode twice.

Adversarial and out-of-scope inputs. What should the feature do when asked something it’s not built for, or something it shouldn’t answer, like a prompt injection attempt, a request for data the user shouldn’t see, or a question entirely outside the feature’s domain? “Doesn’t crash” is the floor; “declines gracefully and doesn’t leak anything” is the actual bar. These cases rarely show up in a demo because nobody adversarially tests their own demo, which is exactly why they need to be in the suite deliberately.

Cost and latency matter alongside correctness. A prompt change that improves accuracy by making the model reason at ten times the length is a real tradeoff, not a pure win, and an eval suite that only checks correctness will never surface that. Track token cost and response time per case alongside the pass/fail, so a “successful” change that quietly triples your inference bill gets caught before it ships, not after the invoice arrives.

How do you judge non-deterministic AI output?

The hard part of AI evals, compared to traditional tests, is that outputs aren’t exact-string-match deterministic: the same prompt can produce two differently-worded but equally correct answers. A few approaches that actually work in practice, usually combined:

• Structural checks wherever the output has structure: the right fields are present, a tool call has valid arguments, a classification lands in the right bucket. This is the cheapest, most reliable check and should cover as much of the suite as possible.
• Rule-based checks for properties you can assert directly, like response length within bounds, required disclaimers present, or no PII in the output.
• LLM-as-judge for genuinely open-ended quality questions, with a judge prompt that’s itself versioned, reviewed, and periodically checked against human judgment, because a judge that’s silently drifted is worse than no judge, since it gives false confidence.

The mix matters more than any single technique. A suite that’s 100% LLM-as-judge is expensive and noisy; a suite that’s 100% exact-match will reject correct answers that are phrased differently and train the team to ignore its failures.

How do you wire an eval suite into CI?

An eval suite that only runs when someone remembers to run it manually will stop running within a month. The suite needs to run automatically on every change that could plausibly affect output (prompt edits, model version bumps, retrieval or tool changes), and the build needs to fail, or at minimum flag a human, when the pass rate or cost/latency profile moves in the wrong direction. That’s the same bar we hold traditional code to, and there’s no good reason to hold AI code to a lower one just because “testing AI is different.” It’s different in technique, not in whether it should happen before you ship.

This is table stakes in every custom AI feature we build now: the eval suite ships alongside the feature, wired into CI, so “did this get better or worse” has a concrete answer on every pull request, not a vibe check after it’s already live.

Two specific ways an eval suite goes wrong even when it exists: see the field guide entries on vibes-based prompt regression and judge prefers its own output.