A task-bound monitor limits injected actions in agents
Treating the Model as an Untrusted Component: Deterministic Policy Enforcement, Human Approval and Taint Tracking Against Prompt Injection in Tool-Using Agents, with Validation on Open-Weights Models · arXiv:2611.11571
In plain words
A tool-using AI can read an email containing an attacker’s instruction and mistake it for the user’s request. This paper tests a security method that treats the AI as untrusted: before a tool call can send a message or change something, a deterministic monitor checks whether the task allows it.
In tests with two open-weights models, the monitor let through fewer attacks than no defense. The models still proposed injected calls often, so the result concerns what the monitor allowed, not whether the models reliably ignored the instructions.
The research
When an agent can act on messages or system settings, an instruction hidden in the data can matter more than whether the model says it will ignore it. These results shift the key security question toward which actions a monitor permits, while the tested models still proposed injected calls often.
The paper introduces and tests a deterministic monitor that authorizes each tool call against the user’s task, even when the language model proposes an injected action.
The team first checked a prompt guardrail on one model and one summarization task: the model emitted an injected call in 20 of 144 samples without it and 18 of 144 with it, a difference the test could not distinguish.
They then ran two Qwen models through 13 tasks, with a monitor checking proposed calls against task permissions.
An effect-based oracle judged whether the attacker’s goal was executed. Compared with no defense, execution fell from 66% to 1% for the 1.5B model and from 86% to 8% for the 7B model.
Because the models still proposed injected calls often, the findings point to action checking rather than reliable model resistance.
The real-model runs covered two open-weights models, static attack families, and a synthetic 13-task suite; the automatic rule may miss paraphrased attempts. Human review and taint tracking were tested only in simulation.
What the evidence shows
- Check the action, not intent
The paper presents a security method for tool-using language-model agents: treat the model as untrusted and use a deterministic monitor to authorize each proposed tool action. The problem is that an agent may read attacker-controlled instructions as data, mistake them for the user’s intent, and then act through its tools.
The key question is whether checking actions can limit harm even when the model proposes an injected call.
- A prompt guardrail showed no detectable effect
The calibration found no detectable change in how often one model emitted an injected call when an in-prompt guardrail was added. This limited test cautions against treating the simulator’s assumed guardrail effect as established for real models.
- The simulated monitor limited harm
In the simulated harness, attack success against a fully compromised model was 23% with D7, the task-bound monitor, compared with 100% for the tested prompt-based guardrail stacks.
This comparison suggests the monitor limited unauthorized side effects under the harness’s assumed model and attacker behavior.
- The monitor filtered proposed calls
The real-model runs show why monitoring actions matters. With no defense, the attacker’s goal was executed in 66% of Qwen2.5-1.5B-Instruct attacks; behind the monitor, it was executed in 1%.
The models still proposed the attacker’s call in many monitored episodes, so the gap reflects the monitor filtering calls, not proof that the model resisted the injected instruction.
- The result depends on model and scope
For Qwen2.5-7B-Instruct, attacker-goal execution was 86% with no defense and 8% behind the monitor. Across both tested models, attacks that passed belonged to the same in-policy semantic-falsification task.
This supports a bounded conclusion: the monitor constrained which proposed actions could pass in these evaluations, while the rate of injected calls the model proposed depended on the model and tested attacks.
Deep
- Original paper authors
- Mustafa S. Aljumaily, Nawar S. Alseelawi, Raghad Alshabandar
- Submitted to arXiv
- License
- arXiv non-exclusive