Deleting a record can leave its secret recoverable by an agent
In plain words
Deleting a record from an agent’s memory may leave behind a summary containing the same information. AfterDeleteBench is a controlled test of whether an agent can still reveal a synthetic secret after its source record is deleted.
The test compared agents that had seen a secret with matched agents that had not. In the main Qwen test, one question recovered the exact secret in about two thirds of exposed profiles, while none of the never-exposed controls did. This result applies to the tested memory setup and secrets.
The research
If someone deletes a record from an agent’s memory, does the agent stop revealing what was in it? A summary or other derived record may remain after the source entry is gone, so the behavioral question is whether the agent can still recover the information.
AfterDeleteBench pairs exposed-then-deleted agents with never-exposed controls to test whether persistent memory still reveals a synthetic secret.
After a record is deleted, can an agent still reveal information it previously saw? AfterDeleteBench tests this question in fixed-weight agents with persistent external memory, where summaries or other derived records can remain after the original entry is removed.
The benchmark compares exposed-then-deleted agents with matched never-exposed controls. In its primary Qwen setting, one direct question yielded exact-reconstruction gaps of 0.680 and 0.640 on two synthetic profile splits, with zero clean-world recovery. This shows exposure-attributable recovery in that setup, including on the generator-held-out split.
These results cover synthetic exact-string targets, controlled external-memory constructions, finite query strategies, and specified response formats.
The paired comparison attributes recovery to exposure in this setup, but does not estimate how much information deletion preserves relative to normal recall or establish behavior in independently designed memory systems.
Analysis
Deep
- Original paper authors
- Ehsan Saleh, Roozbeh Nahavandi, Sama Salarian, Raef Bassily, Andrew Perrault
- Submitted to arXiv
- License
- arXiv non-exclusive