In plain words
Safeguards for AI agents can be tested by asking whether they catch an agent trying to do a harmful side task while doing legitimate work. But attacks run in a separate test setting may not match the tasks and tools of a real deployment.
This paper introduces simulated control evaluations: a language model imitates tool replies using deployment records, while a separate judge estimates whether an attack succeeded.
On LinuxArena, these evaluations predicted safety more accurately than attacks run in BashArena, a different test setting.
The research
If a safeguard is tested on tasks unlike the ones an agent will face, can that test tell us whether it will catch the attacks those tasks permit?
Running attack attempts on live systems risks causing the harms being measured, so the authors ask whether deployment records can make a safer test more relevant.
The authors introduce simulated control evaluations, which use deployment records to simulate tool replies and estimate how well safeguards catch harmful agent actions.
They test this on LinuxArena, where attacks can be executed to establish outcomes.
Researchers compare simulated LinuxArena attacks with attacks executed in BashArena to predict LinuxArena safety, keeping the attack policy, trajectory limit, and monitors the same; the settings still differ in tasks and tool-response realism.
At a 5% audit budget, meaning 5% of honest trajectories are flagged for review, safety prediction error across five monitors was 3.6 percentage points for SCEs and 10.1 for BashArena. Safety measures detection before an attack succeeds without detection. This supports better prediction in this test.
The main accuracy result comes from LinuxArena, and simulated outcomes depend on both the simulator and a separate judge estimating whether attacks succeeded. The study assumes the simulator is trusted.
Analysis
Deep
- Original paper authors
- Jeffrey Olmo, David Zhang, Tyler Tracy
- Submitted to arXiv
- License
- CC BY 4.0