A three-level audit tests when safety cues sway guard decisions

When a guard is judged on labeled examples, how can evaluators tell whether it recognizes unsafe actions or responds to clues in how the examples were written?

A cue that predicts labels may reflect record construction, legitimate safety information, or both, so prediction alone cannot settle whether the guard relies on it.

Shortcut auditing moves from checking whether a cue predicts released labels to testing whether editing it changes an agent-safety guard’s decision.

The authors first tested whether simple cues predicted released labels, then checked whether those patterns held across domains and evaluation units.

They found label prediction could weaken when a whole domain was held out, and that weighting interactions rather than steps changed AgentHarm metric results.

For a more direct test, they edited marker wording and tool names in fixed AgentDojo-Traj records and compared guard decisions with the originals, adjusting for a formatting-only change.

Claude Haiku 4.5’s blocking changed, supporting sensitivity to these representations on that sample; the changes did not reproduce at similar size with Luna.

The paired test used a fixed, non-probability sample of 64 steps from 58 interactions, and the edits were not checked by blinded reviewers for preserving the safety decision.

The effects show sensitivity to these exact edits in one tested guard configuration, not that the cues are generally irrelevant to safety.