Translated requests can change agent guards’ safety verdicts
In plain words
Some software checks an agent’s next tool call and blocks it if the action seems unsafe given the user’s request and what happened earlier. The paper introduces LinguaGuardBench, a test of whether those checks give the same verdict when only the request is translated.
Most open, small and purpose-built guards changed their verdicts more under translation than under a whitespace placebo. For one guard, Nemotron Nano 9B, Amharic requests led it to block fewer steps labelled unsafe than English requests did. The result is specific to the tested guards and steps.
The research
If an agent’s safety check sees a user request in another language, will it still judge the same proposed action the same way? The paper asks whether the request’s language changes a guard’s verdict even when the action and the rest of the labelled step stay fixed.
LinguaGuardBench tests whether translating only a user request changes guards’ verdicts on the same labelled tool-call step.
LinguaGuardBench tests whether a guard’s judgment of a proposed tool call depends on the language of the user’s request.
The authors translated only the request in each labelled step, keeping the history, action, environment information and label fixed, then compared guards’ verdicts across languages.
Most open, small and purpose-built guards changed verdicts beyond their own placebo variation. English rewordings caused smaller changes, supporting a language-specific effect.
For Nemotron Nano 9B, Amharic requests reduced the rate at which it correctly blocked unsafe-labelled steps, showing one concrete consequence of the shifts.
The benchmark reuses labelled steps and does not execute tool calls. Also, the authors’ uncertainty intervals may be too narrow because they group results by trajectory, while translation is applied per instruction.
Analysis
Deep
- Original paper authors
- Hasan Demirkiran, Jens Ernstberger
- Submitted to arXiv
- License
- arXiv non-exclusive