Evals Before You Ship: Why AI Features Need Tests Too

An AI feature without an eval suite in CI turns every prompt tweak, model swap, or provider update into an untested bet. You find out you lost that bet from a support ticket. AI features are software, so they need the same discipline. That means a fixed set of cases with known-good expectations, run automatically, that fails the build on regression. That set must cover the happy path, past incidents, adversarial inputs, and cost and latency. Whether the output looks plausible is not enough.

Traditional software has a straightforward answer for “did this change break anything”: tests, run in CI, on every commit. AI features need that same discipline. The output is harder to grade than an exact-match assertion, but that is not a reason to hold it to a lower bar. Here is what the eval suite needs to cover.

What do eval cases need to cover?

The happy path, precisely. Start with the inputs the feature is built for. Pin down what a correct response looks like for each one. Do not settle for “sounds plausible”: define a specific, checkable property. The right document gets retrieved. The requested field gets extracted correctly. The tool call carries the right arguments. Vague success criteria let an eval suite pass even after quality has dropped with no alert.

The edge cases that already bit you. Every AI feature we have maintained builds a running list of “the time it did something weird.” A malformed input. An ambiguous request. A document with a table that broke extraction. Each one belongs in the eval suite as a regression test, added the same way a bug fix earns a unit test. Skip this step and you fix the same failure mode twice.

Adversarial and out-of-scope inputs. What should the feature do when a user asks something it was not built for? What should it do when a user asks for something it should refuse? A prompt injectionPrompt injection is untrusted input crafted to override a model's system prompt or task. attempt and a request for data the user should not see are two examples. “Does not crash” is the floor. “Declines gracefully and leaks nothing” is the actual bar. Nobody tests their own demo adversarially, so these cases never show up on their own. Put them in the suite on purpose.

Cost and latency matter alongside correctness. A prompt change that raises accuracy by making the model reason ten times longer is a real tradeoff, not a pure win. An eval suite that checks correctness alone will never surface that tradeoff. Track token cost and response time per case next to the pass or fail result. A change that “succeeds” but triples your inference bill should get caught before it ships, not after the invoice arrives.

How do you judge non-deterministic AI output?

Traditional tests check for an exact string match. AI evals cannot: the same prompt can produce two differently worded answers that are both correct. A few approaches work well in practice, usually combined:

• Structural checks wherever the output has structure: the right fields are present, a tool call carries valid arguments, a classification lands in the right bucket. This check is the cheapest and most reliable, so cover as much of the suite with it as you can.
• Rule-based checks for properties you can assert directly: response length within bounds, a required disclaimer present, no personal data in the output.
• LLM-as-judge for open-ended quality questions. Version the judge prompt, review it, and check it against human judgment on a schedule. A judge that has drifted with no alert is worse than no judge, because it produces false confidence.

The mix matters more than any single technique. A suite built entirely on LLM-as-judge costs more and runs noisier. A suite built entirely on exact-match will reject correct answers phrased a different way, and the team will learn to ignore its failures.

How do you wire an eval suite into CI?

An eval suite that only runs when someone remembers to trigger it will stop running within a month. The suite must run automatically on every change that could plausibly affect output: a prompt edit, a model version bump, a retrieval or tool change. The build must fail, or at least flag a human, when the pass rate or the cost and latency profile moves the wrong way. We hold traditional code to that bar already. AI code earns no exception just because grading it takes more thought. The difference is in technique, not in whether the check happens before you ship.

Every custom AI feature we build now ships with its eval suite wired into CI. That gives “did this get better or worse” a real answer on every pull request, not a vibe check once the feature is already live.

An eval suite can still go wrong even after you build it. See the field guide entries on vibes-based prompt regression and judge prefers its own output.