Many agent memory systems lose to direct history search
Memory Tax: Dissecting the Net Value of Agent Memory · arXiv:2610.34073
In plain words
The paper asks whether an agent benefits from turning old conversations into stored summaries or facts, compared with searching the original conversations when a question comes up.
The authors call the difference memory benefit and memory tax: questions the stored memory gets right that direct search misses, and questions it misses that direct search gets right.
Across several tests, many constructed memories answered fewer questions correctly than direct search, though some won particular questions. Further checks suggest memory can lose information while being built or make preserved information harder to find.
The research
When an agent answers questions about earlier conversations, does turning those conversations into organized memory improve on simply searching the original history?
Without that direct comparison, it is hard to tell whether memory processing adds value or whether retrieval alone explains the answer.
The authors compare constructed memories with direct search through raw history, then count memory’s query wins and losses as benefit and tax.
The authors compared constructed memories with direct search across three conversational-memory test sets, using a language-model judge to check answers against references. Many systems scored lower than direct search in many tested conditions, though results varied by system and setting.
Per-question checks found both memory wins and losses. This supports treating memory’s value as a balance: its organization can help, while processing or retrieval can also make useful evidence harder to use.
These comparisons cover the systems, model backbones, and mostly offline English-language tests examined here. Some results use stratified subsets, and the study does not establish how memory performs in long-running, multilingual deployments or with untested memory forms and stronger models.
What the evidence shows
- The question behind memory
The paper is an empirical study of long-term memory for agents. It asks whether processing old conversations into summaries, facts, or other stored units improves answers over searching the raw history directly.
That reference matters because a memory system’s apparent performance alone cannot show whether its extra processing adds value beyond retrieval.
- Compare with direct search
The authors compare representative constructed-memory systems with direct retrieval-augmented generation, which searches minimally processed history for relevant passages. Across three conversational-memory test sets, a language-model judge checks answers against references.
Many memory systems score below direct retrieval in many tested conditions, although some win in particular test and model settings.
- Count wins and losses
A score for all questions can hide opposing outcomes. The authors count queries a memory system answers correctly when direct retrieval fails as memory benefit, and queries direct retrieval answers correctly when memory fails as memory tax.
Their per-query comparison finds both kinds of outcome, so memory is not uniformly better or worse than its reference.
- Trace where losses arise
To locate losses, the authors supply known evidence directly, then separately replay retrieval to see how quickly it recovers stored evidence. These checks point to both information loss during memory construction and difficulty retrieving information that remains.
Related tests beyond conversation also vary by task, supporting a bounded trade-off rather than a general memory advantage.
Deep
- Original paper authors
- Xuan He, Sining Zhoubian, Ruozhen Yang, Yuhao Wen, Zhineng Wang
- Submitted to arXiv
- License
- CC BY 4.0