In plain words
Coding agents sometimes shorten long conversations by replacing them with a summary they use as memory. The paper studies whether this can make text from a file or webpage look like an instruction from the user.
In some tested systems, message roles become text labels such as [User]:, which untrusted text can imitate. Summaries from five such systems often treated a forged approval as the user’s instruction; four systems that kept roles in separate fields did not. A small test with one real agent also showed the false approval could lead to a command being run.
The research
When a coding agent shortens a conversation, it may discard the original messages and use a summary as memory. Can text the agent read from a file or webpage then pass through that summary as if the user had given an instruction?
The paper compares how nine coding-agent harnesses summarize forged user labels, then tests whether a code check can catch false attributions.
The authors tested each harness’s real serializer and compaction prompt with the same forged approval. An LLM judge assessed whether summaries presented it as the user’s standing instruction, and a separate adjudication checked a sample of labels.
Five flattening harnesses had laundering rates of 87–100% per harness; four role-typed harnesses had 0%.
This contrast supports the explanation that, in the tested systems, flattening makes the role label forgeable while typed messages preserve it. A separate live test showed that the false memory could lead to execution in a real session, though only in some poisoned takes.
The live demonstration involved one harness, one model and a small number of takes, so its fractions are not precise estimates.
Most execution tests used a generic agent loop without an approval prompt or operating-system sandbox; they show what the model did in that setup, not what every protected agent would execute.
Analysis
Deep
- Original paper authors
- Arshan Dabirsiaghi
- Submitted to arXiv
- License
- CC BY 4.0