Sparse attention stays close to full attention at long context

Long-context decoding repeatedly reads a growing cache, so selecting fewer positions could reduce the work at each step. But can a method choose individual positions accurately enough to preserve language-modeling performance while also avoiding the overhead of gathering scattered cache entries?

JSSA selects individual context tokens with shared query–key scoring and direct cache reads to preserve attention quality while reducing cache traffic.

The paper asks whether selecting individual context tokens can reduce long-context decoding costs without losing useful attention choices.

Joint Subspace Sparse Attention (JSSA) uses a shared, low-dimensional query–key scoring space to choose positions, then reads their original cache entries directly for attention.

On PG-19, which measures text-prediction quality through perplexity, JSSA was at most 1.11 points from full-cache attention across three models and 32K–128K contexts under the tested cache budgets. This supports near-full-cache language-modeling performance in those conditions.

The results cover specific models, tasks, and hardware setups. The scoring method’s upper bound does not prove that its chosen subspace is globally optimal, and GLM-5.2 has no full-attention checkpoint for comparison.