How do you let an agent act while you keep the ability to undo its mistakes?
Each tool gets the narrowest permission its task needs. A human confirms any action that cannot be undone before it runs. Examples are a refund or an external message. A plausible-looking model decision is never enough on its own. The two controls work together. Least privilege limits the blast radius of anything that slips through. The checkpoint stops the worst actions from running at all.
Also known as approval gate, human-in-the-loop confirmation, least-privilege tool scope
What problem does this solve?#
A model reads one flat stream of tokens. No privileged channel marks some text as trusted instruction and other text as data. Retrieved content enters that same stream. A document or a webpage can carry it. Text that reads like a command gets the same treatment as a real instruction. This holds whether an attacker crafted it or it took that shape by accident. An agent with broad tool access and no gate on irreversible actions has no backstop when that happens.
Each extra tool call widens the set of real systems that a wrong instruction can reach. A refund tool and an email sender are two examples. A delete action is a third. An injection or a bad model decision early in a session stays in context for every later step. The risk builds up across turns.
How does it work?#
Give each tool an explicit allowlist of what it may do. Size that list to the current task. Broad tool access tends to default to everything the agent might ever need, so avoid that default. A narrow grant gives a bad decision a small blast radius by design. The design does not depend on the model's good behavior.
Separately, a human confirms any action that cannot be cleanly reversed before it runs. This applies however confident the agent's reasoning looked. Refunds and permanent deletes fall under this rule. So do external messages and any action where "undo" means manual cleanup. The gate covers only the actions that are expensive to unwind. It does not review the agent's general competence.
Treat model output that feeds a later tool call or another model as untrusted input at that boundary. Validate it as you would validate an anonymous user's request. A successful injection makes it just that. Over time, a specific action type builds a record of correct, low-risk decisions. The team can then relax the checkpoint for that one action. It makes this choice on purpose and leaves the rest of the system gated.
What are the moving parts?#
- Scope every tool to the narrowest permission
Grant what the current task requires. Leave out anything it might need later.
- Classify every tool call by reversibility
At design time, decide which actions are reversible and which need confirmation.
- Gate irreversible actions on human confirmation
Require explicit approval before execution. Apply the rule however confident the agent looked.
- Treat cross-boundary model output as untrusted
Validate output that feeds another tool or model as you would validate an anonymous request.
- Log every confirmation request and its outcome
Record each confirmation request and its outcome as structured data.
- Relax a specific action's gate on purpose
Loosen a checkpoint only after a measured track record for that action type.
When does it fit, and when doesn't it?#
Use it when
- The agent can reach a tool that makes costly or irreversible changes, such as a refund or an external send
- The agent's inputs include retrieved or third-party content next to direct user instructions
- You are early in production rollout and have no track record to justify full autonomy on a given action
- A wrong action costs far more than a brief human delay. The cost can be financial or reputational
Don't use it when
- Every tool the agent can reach is read-only or easy to reverse. No irreversible action needs a gate
- The approval queue has no owner and no SLA. Requests then sit unanswered, which acts as a silent drop. Fix the queue before you add the gate
- The task runs offline or in batch, and no untrusted input reaches the agent. The action is irreversible but low-stakes, such as deleting a scratch file
What does it cost?#
A successful injection or a wrong model decision has a small, bounded blast radius by design
Least-privilege scoping means you build and maintain many narrow tool definitions. One broad, flexible tool would need less upkeep
A plausible-looking but wrong decision cannot trigger an irreversible action on its own
The gate adds latency and an operational queue. Someone must staff the queue and answer each confirmation request
Untrusted treatment of cross-boundary model output catches injected instructions before they reach a second tool
Each hop where one model's output feeds another call needs extra validation logic. That is added engineering work
A deliberate, evidence-based relaxation process lets autonomy grow safely over time
The team must track outcomes per action type to earn each relaxation. A team that relaxes a gate without that data defeats the pattern
How do you know it's working?#
- Confirmation request volume and resolution time per action type, split by outcome
- Share of confirmation requests that reach a human reviewer compared with those that time out
- Reachable-tool audit: which tools each agent instance can call, and whether any exceed the current task's need
- Injection eval pass rate: planted red-team documents should trigger no unconfirmed action
What failure modes does this prevent?#
Sources
- OWASP - OWASP Top 10 for Large Language Model Applications
The ranked risk list our security review checks are organised against. Retrieved 2026-08-24.
- NIST - AI Risk Management Framework (AI RMF 1.0)
The risk-management functions our governance and adoption controls map onto. Retrieved 2026-08-24.
A Ship Audit checks which of these patterns your system needs. It ranks them by what is most likely to break first.