Can a document in our own knowledge base hijack our agent?

What does this look like in production?

• An agent takes an action that traces back to text inside a retrieved document, ticket, or webpage rather than to the user's actual request
• A tool call happens that doesn't correspond to anything the user asked for, immediately following a retrieval or content-fetch step
• A document in the corpus contains phrasing that reads like an instruction ("system note:", "ignore the above", "process this automatically") rather than like content
• A red-team test document planted in the corpus successfully triggers an unintended action when it's retrieved

Why does it happen?

A model reads one undifferentiated stream of tokens. It has no privileged channel that marks some text as trusted instruction and other text as data to summarize. Retrieved content, by definition, enters that same stream. A knowledge base document may contain text that reads like a command. That document might be one you wrote, one a customer uploaded, one scraped from the web. The model has no structural reason to treat it differently from an instruction in the system prompt. To the model, they look the same. An attacker only needs to get text into the corpus. The text will later be retrieved and read in a context where the model has real tool access. A poisoned document that got in through a normal, un-adversarial channel works the same way.

Agents multiply this risk. Every additional tool call is another place an injected instruction can reach something that matters, such as a refund tool or an email sender. A successful injection early in a session persists in context for every later step. It does not need to attack twice.

How do you confirm it's this?

• Plant red-team fixtures (documents containing injection attempts) in a test copy of the corpus, and confirm they don't trigger unintended tool calls when retrieved
• Audit which tools are reachable from any code path that starts with a retrieval or content-fetch step, and list what each one is capable of
• Check whether any irreversible action (refund, delete, send) can be reached without a human confirmation step in between
• Run an injection eval suiteAn eval suite is a versioned set of test cases that measures whether outputs are good. in CI with known attack patterns, and treat any pass-through as a build-breaking regression

How do you fix it?

1. Scope every tool to least privilege

   Give the model the narrowest set of permissions the task needs, with an explicit allowlist. A successful injection then has a small blast radius by construction.

2. Treat model output as untrusted input to the next step

   Anywhere one model's output feeds a tool or another model, validate it the way you'd validate an anonymous user's request. A successful injection makes it functionally exactly that.

3. Require a human checkpoint before irreversible actions

   Require a person to confirm any action that cannot be cleanly undone, such as a refund, a delete, or a message sent externally. Keep that rule at least until the surrounding controls have an established track record.

4. Run injection evals in CI

   Keep a fixed, growing set of known injection patterns and run it automatically on every change. A regression in your defenses then becomes a failed test in a pull request. It is no longer an incident discovered later.

What this doesn't cover

There is no complete model-level fix for prompt injection today. These controls shrink the blast radius of a successful injection. They do not reduce the odds of an injection succeeding to zero. Plan around some attempts getting through the model layer, and make sure what's on the other side of that is bounded, not catastrophic.