Eval Suite (Evals)

Also known as evals, evaluation suite

An eval suite is a repeatable, versioned set of test cases and scoring criteria used to measure whether an LLM system's outputs are actually good — accurate, on-format, safe — before and after every change. Unlike traditional unit tests, evals often score graded or probabilistic quality rather than strict pass/fail, which is why most teams pair automated scoring with LLM-as-judge or periodic human review. Shipping an LLM feature without an eval suite means every prompt or model change is a guess about whether quality went up or down.

← Back to the full glossary