How do you know a prompt change made things better, not just different?
A fixed, human-labelled set of real cases — each with a specific, checkable expected property, not a vibe — runs automatically in CI on every prompt or model change. A drop against the current pass-rate threshold blocks the merge the same way a failing unit test would, so a regression is caught before release, not after a user complaint.
Also known as eval CI gate, frozen regression set, labelled eval suite
What problem does this solve?
A prompt change is a code change with no compiler and no test suite unless a team builds one deliberately. Without a fixed, labelled case set, "did this help" gets answered by re-running the one or two examples the author already had in mind — chosen, consciously or not, because they demonstrate the improvement being made, and therefore neither representative of the full input distribution nor adversarial to the change.
That sample will almost always look like a win regardless of what happens elsewhere in the distribution, so the absence of an eval suite does not make prompt changes safe by default — it makes their risk invisible until a user reports it, at which point the change that caused it may no longer be the most recent thing anyone suspects.
How does it work?
A real eval set is built from 50-100 representative cases pulled from actual production traffic, not invented from imagination, each paired with a specific, checkable expected property — a fact that must appear, a structure the output must have, a thing it must not say — rather than a general "sounds good" impression.
That set runs in CI as a required check on every prompt or model change, the same as a unit test suite, and a drop in pass rate blocks the merge until the regression is fixed or the tradeoff is a deliberate, reviewed decision, not something that ships because nobody happened to notice.
Prompts themselves are version-controlled as code, with the same review process and a paired eval run attached to every diff, so a change always has an author, a diff, and a before/after score, never a silent overwrite in a config panel with no eval attached to it at all.
What are the moving parts?
- Pull 50-100 real cases from production traffic
Source cases from actual usage, not invented examples.
- Attach a specific, checkable property to each case
Define a concrete pass/fail condition, not a vague quality judgment.
- Run the set automatically in CI on every change
Trigger on every prompt or model change, without relying on someone remembering to run it manually.
- Block the merge on a pass-rate regression
Make the eval a required check, the same as a failing unit test.
- Version prompts in source control
Give every prompt change the same review process and diff history as application code.
- Grow the set from real incidents
Every regression that ships becomes a new case in the fixture, so the suite compounds in coverage over time.
When does it fit, and when doesn't it?
Use it when
- Prompt or model changes currently ship on a manual spot-check with no fixed before/after comparison
- The feature is stable enough in scope that a representative 50-100 case set is realistic to define
- More than one person touches the prompts, so an implicit shared understanding of 'good' isn't reliable
- A past regression shipped and wasn't caught until a user noticed
Don't use it when
- The feature is still in early exploration with the shape of correct output changing weekly — the eval set would need rewriting as often as the prompt, which is churn, not safety
- Traffic volume is too low to have 50 representative real cases yet — write a smaller, clearly-labelled placeholder set and plan to expand it, rather than pretend one built from imagination is equivalent
- You already have this and are looking at LLM-as-judge grading specifically — that's a related but distinct pattern to check for judge-preference bias in, not a golden-set gate problem
What does it actually cost?
A regression is caught before merge, with a specific number, not discovered after a user complaint
Building and labelling 50-100 real cases with checkable properties is real, non-trivial work, not a quick script
Prompt changes get the same review discipline as application code
CI run time grows with every added eval case, and the suite needs an owner to keep it maintained as the product changes
A shared, checkable definition of 'good' replaces implicit, individual judgment
A fixed rubric can miss a genuinely new failure mode nobody encoded a case for — passing doesn't mean 'safe in every respect'
The set compounds in value over time as real incidents get folded back in as new cases
A stale set that never gets new cases added quietly stops reflecting current production traffic
How do you know it's working?
- Eval pass rate over time, per feature, tracked as a real metric with history
- CI run time for the eval suite, watched as the case count grows
- Count of production incidents that later became a new eval case, closing the loop
- Share of prompt/model changes merged without a passing eval run, which should be zero if the gate is enforced
What failure modes does this prevent?
A Ship Audit checks which of these patterns your system actually needs, prioritized against what's most likely to break first.