A cache compressor rehearses likely answers before questions arrive
In plain words
A language model keeps a small record of a long text so it can answer questions about it later. If the record is shortened before anyone knows what they will ask, rereading every sentence may waste space on details that are not useful.
Attic-KV is a way to choose what to keep: it rehearses likely answers quoted from the text, along with selected parts of the context. In tests of long-context retrieval and reasoning, it scored higher than whole-document rehearsal at a 3% cache budget on one model and test setting.
The research
If a cache must be compressed before its questions are known, how can it keep the details future answers need without spending its small memory budget rereading every sentence? The paper examines whether changing what the compressor rehearses can direct that budget toward useful information.
Attic-KV compresses a language model’s cache by rehearsing likely quoted answers and a context-adaptive set of anchor positions before future questions are known.
The authors compare Attic’s adaptive anchor rehearsal with a fixed rehearsal amount on Qwen3-8B’s RULER-4K development tasks. With similar average rehearsal amounts, Attic scored 78.0 versus 74.6, supporting the idea that context-specific amounts can help.
On RULER-4K test tasks with 3% of cache entries kept, Attic scored 73.4, above KV 2’s 67.0 and KVzip+’s 31.5. The comparison shows stronger average task performance in this setting, though the methods differ in rehearsal and, for KVzip+, scoring.
The results cover specific models, tasks, cache budgets, and evaluation settings. The main LongBench average excludes two synthetic tasks, and Attic’s single configuration was set using Qwen3-8B development data.
Analysis
Deep
- Authors
- Zhiyun Shi
- License
- arXiv non-exclusive