Token position can weaken retrieval from compressed memory
Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression · arXiv:2609.36322
In plain words
Some long-context language models save memory by compressing tokens into fixed groups before they know what a later question will ask. This empirical study tests whether a fact is easier to retrieve from some positions within a group than others, a pattern called phase sensitivity.
In one 128,000-token test, DeepSeek-V4-Flash-Base's answer counted as right if it began with the correct value.
Accuracy varied across source-key positions; models trained by the authors showed repeating weak positions with compressed memory, while models keeping full attention over tokens stayed comparatively flat.
The research
When a model compresses context before seeing a question, can a high average retrieval score conceal source positions where information is systematically harder to recover? Because compression repeats in fixed windows, equal averages could hide recurring retrieval weak spots tied to token position.
The study measures retrieval across compression-window positions, then uses controlled models, interventions, and theory to examine why weak spots arise.
In a 128,000-token retrieval test, DeepSeek-V4-Flash-Base had to return the value for a key among 16,000 pairs.
The authors counted answers correct if they began with the right value; accuracy differed 40.23 percentage points between the best and worst of eight key-position groups, each with 256 prompts, spanning two cycles of its four-token compression stride.
Qwen3-derived models trained from scratch showed repeating weak positions when they compressed token groups, while models retaining full attention stayed comparatively flat.
One compressed model also varied by over 50 points across positions where an adjacent key and value stayed in one window, so boundary splitting alone cannot explain the pattern.
The DeepSeek position groups shift distractor records along with the target, so that result does not isolate target phase from distractor positions. The Qwen3 comparison reports maxima across different runs rather than matched model pairs.
The mechanistic evidence is limited to tested settings, and the authors did not systematically test whether meaning-preserving shifts affect refusal or compliance in real applications.
What the evidence shows
- A position-dependent retrieval problem
This empirical study asks whether compressing a long context into fixed groups before a question is known makes retrieval depend on where the source information falls within a group. The authors call repeating changes in retrieval by source position phase sensitivity.
They examine existing DeepSeek models, controlled Qwen3-derived models trained from scratch, attention interventions, and an idealized theoretical model.
- A long-context accuracy gap
In a 128,000-token needle-in-a-haystack test, DeepSeek-V4-Flash-Base had to return the value attached to a queried key among 16,000 pairs. The authors counted an answer as correct when it began with the gold value.
Accuracy differed by 40.23 percentage points between the best and worst of eight groups based on the key's position in an eight-token cycle, with 256 prompts per group.
- Compression links to weak positions
To test the role of compression in a controlled model family, the authors trained 23 Qwen3-derived compressed runs and three full-attention baselines from scratch using the same training procedure. The task used 64 random key-value pairs; a hit meant the exact next token for a queried key ranked first across the full vocabulary, measured at 24 source-key positions.
The largest per-run position gap reached 78.0 percentage points among compressed runs and at most 6.1 among baselines. These are group maxima, not matched model pairs.
- Boundaries are not the whole story
A boundary split could make retrieval harder when a key and its adjacent value land in different windows. But in the reference model with non-overlapping eight-token windows, accuracy still differed by more than 50 percentage points across positions 0 through 6, where each pair stayed in one window. Boundary splitting may contribute, but it cannot explain the full pattern.
- Different heads serve different positions
The authors replaced selected query-head outputs with averages from separate calibration prompts balanced across source positions, then compared retrieval accuracy with clean runs. In the full-attention model, important layers reduced accuracy similarly across positions.
In the compressed model, some heads caused losses concentrated at particular positions. Its key compression gates favored early, middle, or late window offsets, while value gates peaked one offset later, matching where those heads mattered most in the intervention.
- Gate shifts support a partial mechanism
After training, cycling gate preferences in five Qwen3-derived models largely shifted weak within-window positions as predicted, but the boundary position stayed weak and overall accuracy fell. In a separate idealized model, loss-minimizing compression selects one position for a key and its successor for the value; under stated assumptions, gate training can make this preference persist. The result offers a possible explanation for learned preferences, not a universal account.
Deep
- Original paper authors
- Xingyu Zhu, Pu, Yi, Ziheng Cheng, Ang Lv
- Submitted to arXiv
- License
- CC BY 4.0