In plain words
Language models can save work by reading only some of the stored information they use to generate text. But ignoring or randomly sampling the unread part can lose important information.
OpenAttention groups unread items and represents each group with a summary that accounts for how many items it contains. On seven long-context test tasks, its average accuracy was close to full attention at a 1% read budget. The result applies to the tested model and tasks.
The research
When a language model reads only part of its stored context, how can it preserve the information in what it skips? Discarding the unread tokens loses information, while sampling can miss high-weight tokens at low read amounts. The paper tests whether compact group summaries can retain more of the attention signal.
OpenAttention clusters cache tokens, reads selected ones exactly, and summarizes the unread groups with count-aware mean proxies to preserve attention information.
The authors tested whether summarizing unread cache tokens could preserve useful attention information better than discarding or sampling them.
They introduced OpenAttention, which clusters tokens and uses estimated cluster error to guide exact reads, while representing unread groups with count-aware mean proxies.
On seven RULER-32K HARD tasks, Llama-3.1-8B-Instruct averaged 86.30 accuracy at a 1% per-query read, including proxy rows, close to dense attention’s 87.04. This comparison supports the method on those tasks and under that read accounting.
These findings cover the tested models, tasks, and read budgets. ErrorGate’s diagonal variance estimate is approximate, not an exact error measure or a certified bound on output error.
Analysis
Deep
- Original paper authors
- Hyunwoo Yu, Jongbeom Lee, Hyunbin Kim, Sungjun Park, Jaemin Choi
- Submitted to arXiv
- License
- CC BY 4.0