EN
Forged labels can turn tool text into user instructions
Compaction Laundering: Provenance Destruction in Context-Compacting Coding Agents · arXiv:2610.29869
Questa pagina non ha ancora una traduzione verificata. È mostrata la versione inglese. Leggi la versione inglese
In parole semplici
Coding agents sometimes shorten long conversations by replacing them with a summary they use as memory. The paper studies whether this can make text from a file or webpage look like an instruction from the user.
In some tested systems, message roles become text labels such as [User]:, which untrusted text can imitate. Summaries from five such systems often treated a forged approval as the user’s instruction; four systems that kept roles in separate fields did not. A small test with one real agent also showed the false approval could lead to a command being run.
La ricerca
When a coding agent shortens a conversation, it may discard the original messages and use a summary as memory. Can text the agent read from a file or webpage then pass through that summary as if the user had given an instruction?
The paper compares how nine coding-agent harnesses summarize forged user labels, then tests whether a code check can catch false attributions.
The authors tested each harness’s real serializer and compaction prompt with the same forged approval. An LLM judge assessed whether summaries presented it as the user’s standing instruction, and a separate adjudication checked a sample of labels.
Five flattening harnesses had laundering rates of 87–100% per harness; four role-typed harnesses had 0%.
This contrast supports the explanation that, in the tested systems, flattening makes the role label forgeable while typed messages preserve it. A separate live test showed that the false memory could lead to execution in a real session, though only in some poisoned takes.
The live demonstration involved one harness, one model and a small number of takes, so its fractions are not precise estimates.
Most execution tests used a generic agent loop without an approval prompt or operating-system sandbox; they show what the model did in that setup, not what every protected agent would execute.
Cosa mostrano le prove
- When memory loses its source
The paper measures provenance laundering in coding-agent harnesses: whether a summary turns untrusted text into something that appears to be the user’s instruction. This can matter when an agent shortens a long conversation and relies on the summary as its memory.
The question is whether losing track of who said what during summarization lets text from a file or webpage gain the authority of a user message.
- Two ways to pass messages
Some harnesses flatten messages into plain text and mark speakers with labels such as [User]:. If untrusted content includes a forged label and the harness does not escape it, the summarizer may treat the following text as a genuine user instruction.
Other harnesses keep the speaker role in a separate field, which text from a file or tool cannot forge. The paper tests this difference using each harness’s real serializer and compaction prompt.
- A sharp split in summaries
The researchers tested the same forged approval in five harnesses that flatten roles and four that keep roles typed. An LLM judge assessed whether each summary recorded the approval as the user’s standing instruction; a separate adjudication checked a stratified sample.
Across the flattening harnesses, 87–100% of summaries were classified as laundering the approval, compared with 0% across the role-typed harnesses.
- When the false memory acts
A false approval can also steer an agent’s plan or lead a review agent to skip a file containing a backdoor. The paper separately tests whether it can lead to command execution.
In a small end-to-end test with the real pi terminal interface, a full chain from refusal before compaction to command execution afterward occurred in three of six poisoned takes and none of five reported clean takes. This shows the effect can reach execution in that setup, while the small sample does not establish a precise rate for other agents.
- A check catches some claims
The paper also tests a code check that compares summary claims attributed to the user with genuine user messages retained by the harness. At one reported setting, it caught 77% of laundering cases and wrongly demoted 2% of legitimate claims in clean summaries.
Some false attributions still passed, so the result supports a partial mitigation rather than a complete fix.
Dettagli
- Autori del paper originale
- Arshan Dabirsiaghi
- Inviato ad arXiv
- Licenza
- CC BY 4.0