The AI terms that matter.

Retrieval-Augmented Generation (RAG) #

Also known as RAG

Retrieval-Augmented Generation (RAG) is an architecture that retrieves relevant documents or passages at query time and feeds them into an LLM's context so it answers from your data instead of its training memory. It trades a training-time knowledge problem for a retrieval-quality problem — chunking, embeddings, and ranking now decide whether the answer is grounded or a plausible-sounding guess. Most "AI doesn't know our data" complaints are RAG pipeline defects, not model limitations.

Embeddings #

Also known as vector embeddings

Embeddings are numeric vector representations of text (or images) positioned so semantically similar content sits close together in vector space. They're the substrate under vector search and RAG: an embedding model converts a query and a corpus into vectors, then similarity search finds the nearest matches. Embedding model choice and dimensionality directly bound retrieval quality — a mismatched or stale embedding model is a common silent RAG failure.

Context Window #

The context window is the maximum amount of text, measured in tokens, a model can consider at once — spanning the prompt, retrieved documents, conversation history, and its own output. A larger window doesn't mean it should be filled: cost scales with tokens processed, and good retrieval still beats brute-force stuffing every relevant document in. Sizing the context window is a cost and latency decision as much as a capability one.

Token #

A token is the basic unit an LLM reads and writes — roughly three-quarters of a word in English, though the exact split varies by model and tokenizer. Usage-based pricing, context window limits, and latency are all denominated in tokens, so token counts are the unit every cost or capacity conversation in this space eventually collapses to.

Model-Agnostic Architecture #

A model-agnostic architecture puts an abstraction layer between your application and any single LLM provider, so you can swap or route between models — Anthropic, OpenAI, open-weight, self-hosted — without rewriting the application. It protects against price changes, deprecations, and capability shifts from any one vendor, and lets different workloads route to whichever model is cheapest or best suited. It's a deliberate design decision, not a default — most teams retrofit it after their first vendor lock-in scare.

Fine-Tuning (vs RAG) #

Fine-tuning further trains a model's weights on your own examples so it changes behavior — tone, format, a narrow skill — baked into the model itself, rather than supplying facts at query time the way RAG does. The two solve different problems: fine-tuning teaches a model how to respond, RAG gives it what to respond with. Reaching for fine-tuning to fix a knowledge or freshness problem that RAG (or better prompting) would solve cheaper and faster is a common build-vs-buy mistake.

Vector Search (Semantic Search) #

Also known as semantic search, similarity search

Vector search finds the nearest matches to a query by comparing embeddings in vector space, rather than matching exact keywords the way traditional full-text search does. It's what lets a RAG pipeline surface a passage about "cancel my plan" when the user typed "how do I stop being billed." In production it's usually paired with metadata filters and a reranker, since raw nearest-neighbor results alone are rarely precise enough to hand straight to the model.

Chunking #

Chunking is splitting source documents into smaller passages before embedding them, so retrieval returns focused, relevant text rather than an entire document. Chunk size and overlap are load-bearing decisions — chunks too large dilute relevance and blow the context budget, chunks too small lose the surrounding context a passage needs to make sense. Most RAG accuracy problems trace back to chunking, not the model.

Agent (Agentic AI) #

An agent is an LLM given a loop, memory, and a set of tools it can call, so it can plan multi-step work and take actions rather than just returning a single answer. "Agentic" describes a pattern along a spectrum — from a single tool call to a fully autonomous multi-step loop — not a strict on/off feature. The engineering that matters is less the model and more the guardrails, observability, and evals around the loop, since an agent that can act is also an agent that can act wrongly.

Tool Calling #

Also known as function calling

Tool calling — also called function calling — is the mechanism that lets a model request a structured action, like calling an API or running a query, instead of only generating text, with the calling application executing the action and returning the result. It's the primitive underneath every agent: an agent is essentially a model given a set of callable tools and a loop to call them in. Reliability here comes down to tight tool schemas and validating what the model actually asks for before executing it.

Model Context Protocol (MCP) #

Also known as MCP

The Model Context Protocol (MCP) is an open standard for connecting LLM applications to external tools, data sources, and other agents through a common interface, rather than every integration being a bespoke one-off. It's the plumbing that lets a single tool or data connector be written once and reused across different agents and applications. Adopting it early is part of building model-agnostic, vendor-neutral agent systems instead of tools wired to one specific assistant — though MCP standardizes the wire format for that connection, not the catalog, contract, or permission decisions behind it.

Context Engineering #

Context engineering is deciding what occupies a model's context window at every step — tool definitions, retrieved content, conversation history, and the task itself — and in what order, not just what to write in a system prompt. It treats the window as a fixed, competed-for resource: every token spent on one slice is a token unavailable to another, so the allocation has to be a deliberate decision rather than whatever's left over once everything else is assembled. Prompt engineering is one input to it, not the whole discipline.

Tool Contract #

A tool contract is the schema a tool exposes to a model — its parameter names, types, required fields, and whatever enums or patterns constrain them. A loose contract lets a model fill an ambiguous field with a plausible-looking guess instead of a real value; a tight one, validated server-side, is what actually stops a hallucinated argument from reaching execution. It's the one piece of documentation a model reads before every call, so it has to carry everything a new hire would otherwise ask about in person.

Idempotency #

An idempotent operation produces the same result no matter how many times it runs with the same input — calling it twice does nothing a single call didn't already do. For a tool that writes, sends, or deletes, idempotency (usually enforced with a unique key passed on every retry of the same logical call) is what makes a retry safe: without it, a timeout followed by an automatic retry can execute the same mutation twice.

Structured Output #

Structured output constrains a model's response to a defined schema — JSON, an enum, a typed object — instead of free-form prose, so the calling code can parse it reliably without brittle regex or string matching. It's foundational to tool calling and agent loops, where a wrongly-shaped response breaks the next step in the chain. Enforcing and validating the schema, not just asking nicely for JSON, is what makes it dependable in production.

Guardrails #

Guardrails are the checks placed around a model's input and output — content filters, schema validation, permission scoping, human-approval gates — that keep an LLM or agent inside acceptable bounds in production. They matter most for agents with real tool access, where an ungrounded or manipulated response can translate directly into an unwanted action, not just a bad chat reply. Guardrails are a design layer added deliberately, not a property models come with by default.

Eval Suite (Evals) #

Also known as evals, evaluation suite

An eval suite is a repeatable, versioned set of test cases and scoring criteria used to measure whether an LLM system's outputs are actually good — accurate, on-format, safe — before and after every change. Unlike traditional unit tests, evals often score graded or probabilistic quality rather than strict pass/fail, which is why most teams pair automated scoring with LLM-as-judge or periodic human review. Shipping an LLM feature without an eval suite means every prompt or model change is a guess about whether quality went up or down.

LLM-as-Judge #

LLM-as-judge is an evaluation technique that uses a — typically stronger or differently-configured — LLM to score another model's outputs against a rubric, at a scale human review can't match. It's useful for grading subjective qualities like tone, relevance, or faithfulness to a source document, but it inherits the judging model's own biases and blind spots, so it's normally calibrated against a smaller human-labeled sample rather than trusted blind. Treat it as one signal in an eval suite, not the whole suite.

Observability #

Observability, in an LLM context, means capturing traces of every prompt, retrieval, tool call, and response so a team can debug why a specific output happened and track quality, latency, and cost over time — not just whether the request returned a 200. Without it, a regression after a prompt or model change surfaces as a vague complaint that "the AI got worse," with no way to pinpoint which step changed. It's the operational counterpart to an eval suite: evals catch regressions before ship, observability catches them after.

Hallucination #

A hallucination is a confident, fluent output that is factually wrong or unsupported by any real source — the model completing a plausible-sounding answer rather than admitting it doesn't know. It's not a bug that gets patched out; it's a structural property of how these models generate text, which is why grounding (RAG) and evals, not a better prompt, are the actual mitigations. A demo that never surfaces a hallucination almost always means the test set was too easy, not that the system is hallucination-free.

Prompt Injection #

Prompt injection is an attack where untrusted input — a document, a webpage, a user message — contains instructions crafted to override a model's original system prompt or task, hijacking its behavior. It matters most once a model can retrieve untrusted content or call tools, since a successful injection can turn a summarization task into an unauthorized action. Defending against it takes input/output evals and guardrails, not just a stricter system prompt, because the system prompt is exactly what's being attacked.

Inference Cost #

Inference cost is what it costs to run a trained model on a request — usually priced per token for hosted APIs — as distinct from training cost, a one-time or periodic expense most teams building on foundation models never pay directly. It scales with token volume, model choice, and context size, and is the line item that turns a good demo into an uneconomical product if it isn't modeled before shipping. Model-agnostic routing — sending easy requests to a cheaper model and hard ones to a stronger one — is one of the most direct levers for controlling it.

Notebook-to-Production #

Notebook-to-production describes the gap between a working data-science notebook or prototype and a system that runs reliably, observably, and cheaply in production — error handling, retries, monitoring, cost controls, and deployment infrastructure a notebook doesn't need. It's usually a bigger lift than the original prototype, which is why it's often underestimated in project timelines. Treating it as a distinct phase, with its own scope and budget, is what separates a demo that ships from one that stalls.

Data Processing Agreement (DPA) #

Also known as DPA

A Data Processing Agreement is the contract that names a vendor as a processor (or subprocessor) of personal data on your behalf, and sets the terms for how they handle it — retention, subprocessors, breach notification, deletion. Routing personal data to a third-party model API almost always brings that provider into this category, whether or not anyone thought of it as "processing personal data" at the time. Privacy and legal reviewers check that the specific model provider and product you're calling is actually named in the DPA, not covered by a generic reference to "AI features."

Data Residency #

Data residency is where data is physically processed and stored, as distinct from where your company or your users are located. A model API call can route through infrastructure in a different region than the rest of your stack, so the residency question has to be answered per provider, not assumed from your primary hosting region. It matters most once a privacy notice, a customer contract, or a framework like GDPR makes a specific residency commitment — at that point residency stops being an infrastructure detail and becomes something reviewers check against what you actually promised.

Red Teaming #

Red teaming is deliberately attacking your own AI system — planting adversarial documents, crafting injection payloads, probing for actions a user shouldn't be able to trigger — to find what breaks before an attacker, or a curious user, finds it first. Unlike a golden-set eval, which checks whether the system gets normal cases right, a red-team suite checks whether it fails safely on cases designed to make it fail. Run once, it is a point-in-time audit; run in CI on every change, it is the same regression protection a golden-set gate gives ordinary quality, applied to security.