DeputyBench evaluates enterprise agents’ contextual permission decisions
DEPUTYBENCH: Evaluating Contextual Safety in Enterprise Agents · arXiv:2611.13355
In plain words
A business request does not always say whether the person asking is allowed to carry it out. The needed facts may be spread across company records and rules, while the agent also needs to finish work that is allowed.
DeputyBench is a set of executable tests that pairs the same business goal with different permission facts and checks what agents do. In a separate test, agents sometimes took restricted actions more often when permission evidence was missing than when the records explicitly showed no permission. The tests do not reveal why an agent acted that way.
The research
A request can describe a legitimate business goal without showing whether its requester is allowed to carry it out. DeputyBench makes the decision hinge on records and policies, and tests both sides of the problem: avoiding prohibited actions and not abandoning legitimate work.
DeputyBench pairs enterprise tasks with different permission facts and audits whether agents avoid prohibited actions while completing legitimate work.
In DeputyBench, company services execute paired tasks with the same business goal but different permission facts. Audits check prohibited actions and legitimate completion.
Across 12 models, 79.1% of counted pairs succeeded in both worlds, showing that avoiding harm alone does not establish appropriate permission handling.
A separate study changed backend authorization facts while keeping the listed request conditions fixed; outcomes tracked those changes. On matched tasks, restricted actions were more frequent with missing evidence than with explicit contrary evidence, though the comparison used distinct protocols.
The tasks were synthetic, in English, and limited to six families and one executor. The scorer still misses some harmful cases and does not cover every disclosure channel, so these rates do not estimate production risk.
What the evidence shows
- Permission is in the records
DeputyBench is an executable benchmark for enterprise agents: it checks whether they use organizational records and policies to decide when to act while preserving legitimate work. A request may state the business goal without identifying the applicable permission checks or where the evidence sits.
- Both worlds count
The benchmark pairs a suspicious task with a legitimate twin that has the same business objective but different deciding permission facts. Company services execute the actions, and task-specific audits check both prohibited actions and legitimate completion.
Success therefore requires appropriate behavior in both worlds, rather than simply stopping whenever permission is uncertain.
- Safety alone falls short
Across 12 models, 79.1% of counted task pairs succeeded in both worlds: the suspicious run avoided audited harm and the legitimate twin completed without audited harm.
This pooled result shows why a high rate of avoiding harm alone cannot establish that an agent handled permission well; legitimate work can be lost too.
- Outcomes follow changed facts
Because the study changed backend permission facts while keeping the listed request conditions fixed, the audits found a large difference between completing authorized tasks and causing unauthorized harm.
This supports outcome dependence on those facts in the tested tasks, but does not reveal the models’ internal process.
- Missing evidence matters
A matched analysis found more restricted actions when permission evidence was unavailable than when contrary evidence was explicit. Much of that difference came from four privacy tasks, identified after the results were inspected.
This points to incomplete evidence coverage as a useful condition to test, but does not establish a shared cause across tasks.
Deep
- Original paper authors
- Hao Zhao, Minghao Lin
- Submitted to arXiv
- License
- arXiv non-exclusive