AutoRedClaw tests attacks on persistent agent state
AutoRedClaw: Automated Red-Teaming for Claw-Style Agents · arXiv:2610.27244
In plain words
AutoRedClaw is a framework that creates security tests for claw-style agents, which can carry information from one task into another. It generates a task, malicious instructions, a small working environment, and a check for whether the attack worked.
Its tests ask whether outside content can poison lasting agent files and whether those files can later prompt a harmful action.
In a fixed collection of 330 cases, OpenClaw and Nanobot met the attack-success conditions in more than 60% of cases on average across four models, in both stages. That result applies to the tested harnesses and collection.
The research
If an agent reloads its memory and configuration across tasks, an instruction encountered once may influence later work. It remains unclear whether security tests can capture both how that instruction enters persistent state and what happens after the agent encounters it again.
AutoRedClaw generates attacks with task-specific environments and success checks, letting researchers test both persistent-state poisoning and its later effects.
AutoRedClaw is an automated red-teaming framework that generates attacks alongside task-specific environments and checks for success. Its two stages test whether external content can poison persistent agent state and whether planted instructions can later cause harmful actions.
In the authors' frozen 330-case collection, attack success rate means the share of cases meeting attacker-success conditions. OpenClaw and Nanobot each averaged above 60% across four models in both stages, evidence that the collection exposes substantial risks for those harnesses.
The findings come from selected models and agent harnesses tested in isolated containers with simulated applications and bounded tasks. They do not establish how other deployments, live services, or prolonged multi-session interactions would behave.
What the evidence shows
- The persistent-state problem
AutoRedClaw is an automated red-teaming framework for testing claw-style agents, which may reload configuration, memory, and reusable skills across tasks. That persistence creates a path from malicious content encountered during one task to instructions that can affect later work.
The authors argue that fixed, manually built tests can miss this transition and may not reflect agents' native workflows.
- Build and qualify attacks
The framework generates benign tasks, attacks, task-specific executable environments, and criteria for detecting success. It prepares only the files and application interactions the task needs.
Before evaluation, checks reject cases without an observable success condition, a reachable attack path, or an initially unsatisfied attack condition; a capable probe agent then tests feasibility. This makes the generated cases assessable, though it does not guarantee success across agents.
- What the tests reveal
RedClaw-Bench is a frozen collection of tests for two stages: whether external content can poison persistent state, and whether planted instructions can cause harmful actions later. Its attack success rate is the percentage of cases meeting attacker-success conditions.
Averaged across four evaluated models, OpenClaw and Nanobot each exceeded 60% on both stages, showing substantial attack success for these harnesses on this collection.
- The signal varies by collection
The authors also compared two models across other agent-security test collections. On RedClaw-Bench's tests of external content poisoning persistent state, both models had higher attack-success rates than on DTap; on AgentDojo and AgentDyn, both were at most 1%.
Because the collections' tasks and conditions are not established as matched, the comparison shows differing evaluation signals rather than proving why they differ.
Deep
- Original paper authors
- Hao Li, Tongyan Hu, Weidi Luo, Shanghao Shi, Xiaogeng Liu
- Submitted to arXiv
- License
- arXiv non-exclusive