Why fixed thresholds can fail as memory stores grow
The Price of Scale in Proactive Memory Recall · arXiv:2610.34507
In plain words
A proactive memory assistant decides whether to bring up something from an earlier conversation, even when nobody asks. As its store grows, comparing a message with more memories makes an irrelevant one more likely to look like a match.
The paper explains this effect and introduces a gate that compares a message’s best match with matches for other messages. In replay tests on LongMemEval, this gate kept false alarms near 5% across tested store sizes, with hit rates close to a threshold tuned using labels.
The research
When an assistant brings up an old detail without being asked, how can it avoid increasingly frequent interruptions as its memory store fills? The paper asks how false alarms and successful recalls change with store size, and whether a gate can control false alarms without labeled examples.
The paper models how store growth affects false alarms and proposes a label-free cue-swap gate to control them when reference cues are exchangeable.
The authors use question-and-history replay datasets to measure whether a gate surfaces a relevant memory or speaks when none is relevant. Their theory says that holding the overall false-alarm rate steady as a store grows requires each memory comparison to be more selective.
In LongMemEval-M, a threshold tuned on eight memories instead produced 87% false alarms at 2,048 for cosine scoring.
A cue-swap gate sets its threshold by comparing the current cue’s best score with scores from other cues, without label tuning. Across four LongMemEval settings, its false-alarm rates stayed between 2.8% and 5.9%, and its hit rate was within a few points of a label-tuned z-score threshold.
This supports the method under those replay conditions, while LoCoMo shows that its result depends on the reference cues.
The cue-swap guarantee depends on reference cues being exchangeable with the current cue. The replay tests use benchmark questions and model-written cues rather than real users’ messages, and all experiments are in English.
What the evidence shows
- A gate faces more comparisons
The paper studies proactive memory recall: software that decides whether to bring up a stored memory without being asked. Each new message is compared with every memory, and the gate speaks if the best score clears a threshold.
The central question is how to keep false alarms controlled as the number of stored memories grows, while still surfacing relevant ones.
- Fixed thresholds lose control
The authors replay LongMemEval and LoCoMo, datasets pairing questions with conversation histories, using user turns as memories and questions as cues. In a LongMemEval-M replay with explicit questions and bge-small embeddings, cosine and tf-idf thresholds were each set to 5% false alarms at eight memories.
With those thresholds held fixed, false alarms on irrelevant-memory trials rose to 87% and 100%, respectively, at 2,048 memories.
- The theory tracks hit rates
The theoretical analysis explains the pattern under assumptions about independent irrelevant-memory scores and an independent relevant-memory score.
To keep the overall false-alarm rate at α across N irrelevant memories, each comparison must have a false-positive probability of about −ln(1−α)/N. The analysis’s predicted hit rate for an oracle cosine gate tracked observed rates within 0.9 percentage points across the tested LongMemEval-M store sizes.
- Calibration trades noise for recall
Per-cue z-score normalization can retain more hits than cosine scoring at approximately matched false-alarm rates. In the explicit LongMemEval-M replay with bge-small at 2,048 memories, the label-tuned z-score gate hit 18% of positive trials, compared with 8% for the oracle cosine gate.
The cue-swap gate then offers a way to set a threshold without label tuning; its results depend on whether reference cues are comparable to the current cue.
Deep
- Original paper authors
- Yichuan Chen, Xin Wang
- Submitted to arXiv
- License
- arXiv non-exclusive