Evaluation
Eval Suite (Evals)
Also known as evals, evaluation suite
An eval suite is a versioned set of test cases that measures whether outputs are good.
It checks whether outputs are accurate, on-format, and safe, before and after every change.
Unlike unit tests, evals often score graded or probabilistic quality rather than strict pass or fail.
Most teams pair automated scoring with LLM-as-judge or periodic human review.
Shipping a change without an eval suite makes every quality claim a guess.
← Back to the full glossaryRelated terms
Agent (Agentic AI) An agent is an LLM given a loop, memory, and tools it can call. Fine-Tuning (vs RAG) Fine-tuning trains a model's weights on your own examples, changing its behavior directly. Hallucination A hallucination is a confident, fluent output that is factually wrong or unsupported. LLM-as-Judge LLM-as-judge uses another LLM to score a model's outputs against a rubric, at scale.
More in Evaluation
Observability Observability captures traces of every prompt, retrieval, tool call, and response for debugging. Prompt Injection Prompt injection is untrusted input crafted to override a model's system prompt or task. Shadow Deployment A shadow deployment runs a candidate release against real production traffic, offline, before release.