Agents do better when memory lessons link back to source runs
Don't Just Trust Your Memory: Evidence-Linked Memory Reduces Memory-Induced Degradation in LLM Agents · arXiv:2611.14320
In plain words
Agents can remember lessons from earlier tasks, but a success-or-failure label does not show which action caused the result. A mistaken lesson can steer an agent away from a task it could otherwise solve.
WikiTrace links each short lesson to a summary and the original task record. The agent can inspect that record when a lesson is unclear or conflicts with another. Across eight task suites, WikiTrace scored higher on average than giving the agent the same lessons without their source records.
The research
A remembered rule can make an agent fail a task it could solve, and a success-or-failure label alone may not show whether the rule fits. Linking lessons to their source records gives the agent a chance to check the guidance before acting. In these tests, that approach scored higher than using the same lessons without source access.
WikiTrace links each agent memory lesson to its source run, so the agent can inspect evidence before relying on guidance that may be wrong or incomplete.
In the main evaluation, methods used the same agent and test tasks, with memory built once from no-memory training runs and three trials per suite.
WikiTrace averaged 90.2 points across eight suites, compared with 87.0 for Insights Only, which used the same lessons without source records or an instruction to inspect them.
Because that comparison adds source access and the inspection prompt together, it cannot separate their effects. The higher average supports the combined approach in these tests.
The failure diagnoses come from an LLM judge’s classifications, and linked runs do not verify that a lesson is correct. The evaluation covered five task tests and eight suites, with memory built once and kept fixed during testing.
What the evidence shows
- Memory needs a way to check
WikiTrace is an evidence-linked memory system for language-model agents that reuse past task runs. The evaluation tests agents on simulated text-world tasks and tasks involving software tools; each test reports its own task-performance score, and the study averages those scores across eight suites.
The central question is whether an agent can reuse concise lessons while checking the records behind them when a lesson may mislead it.
- Some failures follow a lesson
The study examines tasks solved in all three no-memory trials but failed with memory. An LLM judge compares each failure with a no-memory run of the same task and classifies the first divergence.
It attributes 52% of the examined cases to a followed memory item, including 20% to an item it judged wrong. This diagnosis suggests why checking a lesson’s source could matter, while remaining a judge’s classification rather than a verified cause.
- Lessons point back to runs
WikiTrace links each concise lesson to a summary of its source run and to the original action-and-observation record. The agent starts with the lessons and is prompted to inspect those records when a lesson lacks detail, comes from a failed run, conflicts with another lesson, or an action fails.
The links provide a way to examine the evidence a lesson leaves out; they do not establish that the lesson or summary is correct.
- Scores rise across the suites
Across eight task suites, WikiTrace averages 90.2 points, compared with 79.5 for No Memory, and has the highest score on each suite. The tests cover simulated text worlds and software-tool tasks, and the reported scores use each test’s original task-performance measure.
The average advantage shows that adding this memory system coincided with higher scores under the study’s fixed-memory evaluation procedure.
- The same lessons, with sources
Across eight task suites, WikiTrace averages 90.2 points versus 87.0 for Insights Only, which gives the same lessons without source records or a prompt to inspect them.
This comparison supports the combined addition of linked records and inspection prompts in these tests, but cannot separate their effects.
Deep
- Original paper authors
- Yukyeong Lee, Yumin Choi, Jinheon Baek, Sung Ju Hwang
- Submitted to arXiv
- License
- CC BY 4.0