Memory systems struggle to turn user history into sound choices
HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memory · arXiv:2610.24520
In plain words
HABIT-Bench is a test of whether an AI assistant’s memory can help it use someone’s habits correctly. A past choice may apply only in a certain situation, and a suggestion the person never accepted should not count as their preference.
The test gives an answer model a request, four possible actions, and context from a memory system, then checks whether it picks the valid action.
In one tested setting, the best memory method scored about the same as a method that retrieved past sessions; a compacted-history control scored higher than the memory methods.
The research
When an assistant has a long interaction history, how can it tell whether a repeated choice is a lasting default, a context-specific exception, or only a suggestion the user never accepted? The history may contain clues without making the right later action obvious.
HABIT-Bench tests whether memory systems infer user habits from incomplete histories and apply them to later choices.
Can an agent infer a user’s behavioral default from a long history and know when it applies? HABIT-Bench tests this with synthetic histories and later requests where context, exceptions, and whether a user accepted a suggestion matter.
Memory methods supply context to an answer model, which chooses among four actions; the evaluator scores exact matches to the valid choice. In the GLM-5.3-Flash setting, the best memory method was close to session retrieval, while a compacted-history control scored higher than all eight memory methods.
This supports the authors’ view that elaborate memory pipelines do not consistently improve these decisions.
The histories are controlled synthetic lifelines, not longitudinal records of the people whose task dialogues supplied the seeds.
Travel is a diagnostic set, not an independent generalization test, and differing context construction and budgets mean the comparisons assess complete pipelines rather than isolating a memory architecture.
What the evidence shows
- A history is not a habit
HABIT-Bench is a benchmark for evaluating whether language-model agents can infer behavioral defaults from incomplete evidence across sessions and apply them to later requests.
The question matters because past interactions can be ambiguous: a repeated choice may be context-specific, and an assistant’s suggestion does not become a user preference just by appearing in the record.
- How choices are tested
The benchmark builds synthetic histories in four domains and asks a shared answer model to choose one of four actions for each request using context supplied by a memory method. The evaluator checks the choice against the uniquely valid action.
Standard methods see only the history available at that point, not private habit annotations, future sessions, or the gold answer.
- Richer memory did not lead
In the GLM-5.3-Flash setting, A-MEM, the strongest memory method, scored 33.61% macro accuracy, close to Dense RAG’s 33.21%. Macro accuracy is the unweighted mean of accuracy across the four domains.
The Full Memory control, which supplied a compacted history, scored 41.98%, above all eight memory methods in that setting.
- Evidence still leaves work
A diagnostic gave the answer model private annotated evidence or, in a separate condition, the private habit state and required action. With Qwen3-235B-A22B, scores were 39.26–58.25% across domains with annotated evidence and 98.41–99.26% with the habit state and action.
The contrast suggests that finding evidence and using it to reconstruct a valid decision are distinct challenges, though the conditions also differ in the information supplied.
Deep
- Original paper authors
- Junming Zhang, Rubin Wei, Xu Cheng, Hao Sun, Jiarui Wang
- Submitted to arXiv
- License
- arXiv non-exclusive