Evaluation
Prompt Injection
Prompt injection is untrusted input crafted to override a model's system prompt or task.
It matters most once a model can retrieve untrusted content or call tools.
A successful injection can turn a summarization task into an unauthorized action.
Defending against it takes input and output evals and guardrails, not a stricter system prompt.
The system prompt cannot defend itself, because it is exactly what is being attacked.
← Back to the full glossaryRelated terms
Guardrails Guardrails are the checks that keep an LLM or agent inside acceptable bounds in production. Tool Poisoning Tool poisoning hides an instruction in a tool's description that the model reads as legitimate. Hallucination A hallucination is a confident, fluent output that is factually wrong or unsupported. Red Teaming Red teaming is deliberately attacking your own AI system to find what breaks first.
More in Evaluation
Eval Suite (Evals) An eval suite is a versioned set of test cases that measures whether outputs are good. LLM-as-Judge LLM-as-judge uses another LLM to score a model's outputs against a rubric, at scale. Observability Observability captures traces of every prompt, retrieval, tool call, and response for debugging. Shadow Deployment A shadow deployment runs a candidate release against real production traffic, offline, before release.