# Eval Suite (Evals)

Source: https://customlabs.io/glossary/eval-suite/
Updated: 2026-09-13

Evaluation

# Eval Suite (Evals)

Also known as evals, evaluation suite

An eval suite is a versioned set of test cases that measures whether outputs are good.

It checks whether outputs are accurate, on-format, and safe, before and after every change.

Unlike unit tests, evals often score graded or probabilistic quality rather than strict pass or fail.

Most teams pair automated scoring with LLM-as-judge or periodic human review.

Shipping a change without an eval suite makes every quality claim a guess.

[Insight: Evals before you ship](https://customlabs.io/insights/evals-before-you-ship/)[AI Readiness Scorecard](https://customlabs.io/tools/ai-readiness/)

[← Back to the full glossary](https://customlabs.io/glossary/)

## Related terms

[Agent (Agentic AI) An agent is an LLM given a loop, memory, and tools it can call.](https://customlabs.io/glossary/agent/)[Fine-Tuning (vs RAG) Fine-tuning trains a model's weights on your own examples, changing its behavior directly.](https://customlabs.io/glossary/fine-tuning/)[Hallucination A hallucination is a confident, fluent output that is factually wrong or unsupported.](https://customlabs.io/glossary/hallucination/)[LLM-as-Judge LLM-as-judge uses another LLM to score a model's outputs against a rubric, at scale.](https://customlabs.io/glossary/llm-as-judge/)

## More in Evaluation

[Observability Observability captures traces of every prompt, retrieval, tool call, and response for debugging.](https://customlabs.io/glossary/observability/)[Prompt Injection Prompt injection is untrusted input crafted to override a model's system prompt or task.](https://customlabs.io/glossary/prompt-injection/)[Shadow Deployment A shadow deployment runs a candidate release against real production traffic, offline, before release.](https://customlabs.io/glossary/shadow-deployment/)