Can a document in our own knowledge base hijack our agent?

What does this look like in production?

• An agent takes an action that traces back to text inside a retrieved document, ticket, or webpage rather than to the user's actual request
• A tool call happens that doesn't correspond to anything the user asked for, immediately following a retrieval or content-fetch step
• A document in the corpus contains phrasing that reads like an instruction ("system note:", "ignore the above", "process this automatically") rather than like content
• A red-team test document planted in the corpus successfully triggers an unintended action when it's retrieved

Why does it happen?

A model reads one undifferentiated stream of tokens — it has no privileged channel that marks some text as 'trusted instruction' and other text as 'data to summarize.' Retrieved content, by definition, enters that same stream. If a document in your knowledge base — one you wrote, one a customer uploaded, one scraped from the web — contains text that reads like a command, the model has no structural reason to treat it differently from an instruction in the system prompt, because to the model, they look the same. An attacker (or a poisoned document that got in through a normal, un-adversarial channel) only needs to get text into the corpus that will later be retrieved and read by the model in a context where it has real tool access.

Agents multiply this risk specifically, because every additional tool call is another place an injected instruction can reach something that matters — a refund tool, an email sender, a delete action — and a successful injection early in a session persists in context for every later step, so it doesn't need to attack twice.

How do you confirm it's this?

• Plant red-team fixtures in a test copy of the corpus — documents containing injection attempts — and confirm they don't trigger unintended tool calls when retrieved
• Audit which tools are reachable from any code path that starts with a retrieval or content-fetch step, and list what each one is capable of
• Check whether any irreversible action (refund, delete, send) can be reached without a human confirmation step in between
• Run an injection eval suite in CI with known attack patterns, and treat any pass-through as a build-breaking regression

How do you fix it?

1. Scope every tool to least privilege

   Give the model the narrowest set of permissions the task genuinely needs, with an explicit allowlist, so a successful injection has a small blast radius by construction rather than broad access it's merely trusted not to misuse.

2. Treat model output as untrusted input to the next step

   Anywhere one model's output feeds a tool or another model, validate it the way you'd validate an anonymous user's request — because a successful injection makes it functionally exactly that.

3. Require a human checkpoint before irreversible actions

   A refund, a delete, a message sent externally — anything that can't be cleanly undone should have a person confirm it, at minimum until the surrounding controls have an established track record.

4. Run injection evals in CI

   Keep a fixed, growing set of known injection patterns run automatically on every change, so a regression in your defenses is a failed test in a pull request, not an incident discovered later.

What this doesn't cover

There is no complete model-level fix for prompt injection today — these controls shrink the blast radius of a successful injection, they do not reduce the odds of an injection succeeding to zero. Plan around some attempts getting through the model layer, and make sure what's on the other side of that is bounded, not catastrophic.