Why did quality drop after a prompt tweak nobody thought was risky?

What does this look like in production?

• A prompt or model change ships after "looking good" in a manual spot-check, with no fixed test set run before or after
• Quality complaints rise after a change that the team considered minor or purely stylistic
• Nobody can answer "did this get better or worse" with a number — only with an impression
• The same class of failure gets fixed, ships fine for a while, then quietly reappears after an unrelated later change

Why does it happen?

A prompt change is a code change with no compiler and no test suite unless the team builds one deliberately. Without a fixed, labelled set of cases and expected properties, "did this help" gets answered by re-running the one or two examples the author already had in mind — examples chosen, consciously or not, because they demonstrate the improvement being made. That sample is neither representative of the full input distribution nor adversarial to the change, so it will almost always look like a win, regardless of what happens elsewhere in the distribution.

The absence of an eval suite doesn't make prompt changes safe by default — it makes their risk invisible until a user reports it days or weeks later, at which point the change that caused it may no longer be the most recent one anyone suspects.

How do you confirm it's this?

• Check directly: does a labelled eval set exist for this feature at all, anywhere in the repo or CI config?
• If one exists, check whether prompt and model changes are actually required to run it before merge, or whether that step is optional and often skipped
• Look at how the last five prompt changes were reviewed — was there a documented before/after score, or only a description of intent?
• Sample recent quality complaints and check whether any correlate in time with an un-evaled prompt change

How do you fix it?

1. Build a real eval set: 50-100 representative cases

   Pull real cases (not invented ones) with a specific, checkable expected property each — not "sounds good," but a concrete pass/fail condition per case.

2. Run it in CI on every prompt or model change

   Make the eval suite a required check, not an optional script someone can forget to run, so a regression is caught before merge instead of after a complaint.

3. Block the merge on regression

   Treat a drop in the eval pass rate the same way you would treat a failing unit test — the change doesn't ship until it's fixed or the regression is a deliberate, reviewed tradeoff.

4. Version prompts as code

   Store prompts in version control with the same review process as application code, so every change has a diff, an author, and a paired eval run, not a silent overwrite in a config panel.

What this doesn't cover

An eval suite catches regressions its cases exercise — it will not catch a failure mode nobody thought to write a case for. Treat a clean eval run as evidence the known risks didn't regress, not as proof the change is safe in every respect.