Prompt Injection Is a Data Problem: A Threat Model You Can Ship Against

Prompt injection is untrusted data reaching a model that has no reliable way to tell instructions from content. Text crafted to look like a command gets executed like one, no matter where it came from. You cannot filter your way out of this. The model’s confusion is the vulnerability, not the wording of any one payload. The defense that holds is architectural: least-privilege tools, treating every model output as untrusted, and a human checkpoint before anything irreversible happens.

Every LLM security conversation arrives at the same instinct: block the bad phrases, strip “ignore previous instructions,” ship it. That instinct is why most teams’ first defense fails within a week of going live. Prompt injection isn’t a string-matching problem. It’s a consequence of how the model reads text, and it needs a threat model, not a blocklist.

Why can’t you just filter out malicious prompts?

A large language model receives one undifferentiated stream of tokens. It has no separate, privileged channel for developer instructions versus data to summarize. Both arrive as text in the same context window. The model infers intent from content, not from a trusted origin flag. That is the entire vulnerability: anything that reads like an instruction can act like one. It does not matter whether it came from your system prompt or from a paragraph buried in a PDF the model was asked to summarize.

Filtering blocks phrases such as “ignore previous instructions” or “you are now,” hoping to catch the attack in the wording. The defender always loses this cat-and-mouse game, because the attacker does not need those exact phrases. A payload can be base64-encoded and paired with an instruction to decode it. It can be translated into a language the filter does not cover. It can also be split across retrieved chunks that only combine into an attack once they are all in context. Or it can just be rephrased until it slides past whatever pattern got blocked last week. None of that takes sophistication. It takes noticing that the filter matches strings, not intent, and intent has infinite phrasings.

Where does untrusted text enter your system?

The attack surface is every place your system reads content it did not author. It then hands that content to a model that acts on what it reads:

• RAG documents: anything indexed from a source you do not fully control, such as a wiki, a shared drive, or a customer-uploaded file.
• Tool and API outputs: a third-party response, a database record a customer wrote, a webhook payload.
• Fetched web pages: any browsing or scraping tool exposes the model to whatever the page’s owner put there.
• Inbound email and support tickets: text a stranger wrote specifically to be read by whatever processes it.
• Uploaded files: PDFs, spreadsheets, images with embedded or OCR’d text.
• The agent’s own prior transcript: a successful injection early in a session persists as context for every later step. It does not need to attack twice.

Agents and tool-calling do not introduce a new category of risk here. They multiply the existing one. Every added tool call is another place untrusted text can enter the loop. An agent that reads and acts across many steps gives an injected instruction more chances to reach something that matters before anyone notices.

What can an attacker do?

Once an instruction lands, the impact tracks whatever the model is authorized to do next. An attacker can exfiltrate data by having the model include secrets or private records in its output. Or the model can send them in a tool call to an attacker-controlled endpoint. They can trigger unauthorized tool calls: the “confused deputy” pattern, where the model uses its own legitimate permissions on the attacker’s behalf. They can leak the system prompt or embedded credentials back to whoever asked for them. Or they can poison retrieval: write a document meant to be retrieved later, so it injects then instead of attacking the current request.

Here is a concrete version of the pattern. A support agent with a tool to issue refunds and send email reads an incoming ticket. The ticket’s body, not its subject line, carries a fake instruction. It claims the customer is owed a full refund and a formal apology, and tells the agent to process both now. Suppose the agent treats that ticket content as plain data to summarize, not as untrusted input sharing its own context window. It still issues the refund and sends the email. Nothing about the transaction looks anomalous from the outside. It is the same tool call the agent makes correctly a hundred times a day.

How do you ship against it?

No single control closes this gap, which is exactly why defense-in-depth is the right shape. It is the same shape a security reviewer expects to see when this system comes up for review:

• Least-privilege tool scopes. Give the model the narrowest set of tools and permissions the task needs. Use an explicit allowlist rather than broad access it is trusted not to misuse. A refund tool that can only issue refunds under a cap bounds the blast radius of a successful injection by construction. It cannot also send arbitrary email.
• Treat every model output as untrusted input to the next step. If one model’s output feeds another model, a tool, or a downstream system, validate it. Treat it the way you would treat a request from an anonymous user. Functionally, that is what an injected output is.
• Human-in-the-loop before irreversible actions. A refund, a delete, an email sent to a customer: anything that cannot be cleanly undone should have a checkpoint a person confirms. Keep that checkpoint at least until the surrounding controls have a track record.
• Separate trusted and untrusted context. Where the architecture allows it, keep instructions and untrusted content out of one undifferentiated prompt. A dual-LLM or quarantine pattern shrinks the surface a single injection can reach. One model handles untrusted content and can only pass structured, constrained data to the privileged model.
• Guardrails and output validation. Run structural checks and rule-based validation on what the model produces. Do not just trust that it produced something.
• A red-team injection eval suiteAn eval suite is a versioned set of test cases that measures whether outputs are good. in CI. The same discipline we argue for evals generally applies here. Run a fixed set of known injection patterns automatically on every change. A regression in your defenses then becomes a failed test, not a surprise.
• Step-level tracing. If you cannot trace which step in a chain acted on injected content, every incident becomes archaeology. ObservabilityObservability captures traces of every prompt, retrieval, tool call, and response for debugging. at each step turns “the agent did something wrong” into a specific, findable defect.

What this doesn’t solve (honest limits)

There is no complete, model-level fix for prompt injection today, and any claim otherwise oversells it. Defense-in-depth shrinks the blast radius of a successful injection. It does not reduce the odds of an injection succeeding to zero. Any control here can in principle be bypassed by an attacker willing to iterate. Models are getting measurably better at resisting some injection patterns. But “the model is more robust this quarter” is not a control you can point to in an audit. Least privilege, human checkpoints, and evals are controls you own, regardless of which model you run. Plan around the assumption that some injection attempts will get through the model layer. Make sure the other side of that is bounded, not catastrophic.

Where to start

Suppose you run an LLM system that reads anything it did not author: a document, a ticket, a web page. Start by mapping which of its tools could do damage if hijacked. Check whether a human sits between the model and the irreversible ones. That is exactly the gap our readiness & diligence review is built to surface before it becomes an incident. If you want a second set of eyes on an existing agent’s or RAG system’s actual attack surface, we are glad to help pressure-test it.

One of the most common vectors is a document in your own corpus turning against you. See the field guide entry on injection via retrieved content for the mechanics.

This threat model argues for specific controls: least-privilege scoping, a human checkpoint, a red-team suite in CI. The AI governance layer expects a system cardA system card is a short document stating an AI system's purpose, limits, and owner. to state these up front. They should not still be a live question once the system already handles production traffic.