A trust gate cuts attacks on Haiku, with mixed results across models
SentryBench: A Cost-Sensitive Benchmark for Adversarial Robustness of Tool-Using LLM Agents in Public-Safety Deployments · arXiv:2610.34535
In plain words
A public-safety agent that uses tools can make two kinds of harmful mistake: ignore a real emergency or take an unjustified action. SentryBench is a test framework that checks both, using emergency scenarios, attacks, and a computer sandbox with pretend tools.
In tests of three models, a blanket instruction to be cautious often led to missed emergencies. The targeted trust-gating wrapper lowered attack success on Haiku; GPT-5 already had zero attack success with no defense.
It did not increase emergency refusals across the tested models; provider filters and the small number of GPT-5 attacks that ran limit that result.
The research
When an agent can dispatch help or send alerts, refusing an instruction is not automatically safe: missing a genuine emergency can also cause harm. Can a defense reduce attacks without making agents less likely to act on urgent cases?
SentryBench measures missed emergencies alongside attacks, and tests a targeted trust gate: attack success fell on Haiku, stayed at zero on GPT-5, and was unchanged on Sonnet, without higher refusals.
The authors ask whether public-safety agents can resist attacks without becoming so cautious that they miss real emergencies. SentryBench tests both errors, and the authors compare a targeted trust gate with no defense and with a prompt-only caution instruction.
Caution changed attack success by at most 3.7 percentage points, but over-refusal reached 100% for Sonnet and GPT-5. The trust gate lowered attack success from 2.7% to zero on Haiku; GPT-5 was already at zero with no defense.
It kept over-refusal at or below baseline across all three models, supporting a targeted approach in these tests.
These results come from a seeded sandbox with mocked tools, three models, and a limited set of scenarios. Provider filtering reduced the number of GPT-5 attacks that actually ran, so its reported zero attack success does not describe those blocked prompts.
What the evidence shows
- Safety has two error directions
SentryBench is a benchmark and evaluation framework for tool-using language model agents in public-safety scenarios. It tests whether agents respond to genuine emergencies while resisting attacks that try to redirect their actions.
The central question is whether reducing risky actions by telling an agent to be cautious also makes it fail to act when help is warranted.
- Measure both kinds of harm
The framework uses urgent benign scenarios to measure failures to act, ambiguous benign scenarios to measure unwarranted actions, and adversarial scenarios to measure attack success. A deterministic scorer reads tool-call logs.
Each rate is calculated over its relevant executed scenario group, while mean harm combines the costs of both error types across the full scenario set.
- Caution can miss emergencies
In comparisons with no defense, a prompt-only caution instruction changed attack success by at most 3.7 percentage points. But over-refusal reached 100% for Sonnet and GPT-5, and caution had the highest mean harm for all three tested models. This evidence shows why fewer attacks alone cannot establish that a defense reduced harm.
- A targeted gate changes the tradeoff
The authors also tested a trust-gating wrapper, which checks untrusted content for manipulation and routes consequential tool calls to a human when the check triggers. Compared with no defense, attack success fell from 2.7% to zero on Haiku; GPT-5 was already at zero.
Over-refusal stayed at or below baseline on all three models. On Sonnet, attack success and over-refusal matched baseline, but containment increased.
- Filtering narrows the evidence
The GPT-5 model-level zero is based on as few as 12 executed adversarial episodes because Azure’s content filter blocked 11 to 15 prompts per defense condition before the model ran. The filter also blocked some genuine emergencies, so the authors report model and system outcomes separately.
The finding is bounded by what ran in this sandbox and cannot resolve outcomes for filtered prompts.
Deep
- Original paper authors
- Md Abu Yusuf, Oyshi Tabassum Aditi, Md Faysal Hosain Mazed
- Submitted to arXiv
- License
- CC0