One shared cache representation serves several model layers
BellowsKV: One Latent for Many Layers, Expanded on Demand · arXiv:2610.35023
In plain words
Language models keep records of earlier words so they can answer using a long input, but every layer stores its own cache.
BellowsKV is a compression method that needs no model retraining: it stores one representation per token for nearby layers, then lets each layer rebuild its own attention keys and values when needed.
The authors found neighboring layers’ inputs were similar even when their cache entries were not. Across three models, the quantized version kept about 97–98% of the uncompressed models’ scores on average over 11 long-context tasks, at roughly 13–14 times compression.
The research
A long-context language model keeps a separate record of past tokens in every layer, and each request needs its own cache. Can adjacent layers share that memory during generation while still recovering the different attention keys and values each layer needs?
BellowsKV is a training-free method that stores a calibrated shared representation for each token across adjacent layers, then reconstructs each layer’s separate keys and values.
Three instruction models were tested with the xKV authors’ harness on LongBench, 11 English tasks involving question answering, summarization and finding details in long text. For each task, the harness score was divided by the same model’s dense-cache score, then averaged.
BellowsKV’s quantized version retained 97–98% at 13–14x compression; xKV retained 54–84% near 16x. This supports storing the shared representation at full size with fewer bits, although the compression rates differ.
The LongBench results cover three models and 11 tasks, with inputs longer than 4,000 and within 32,000 tokens. BellowsKV’s comparison points also use lower compression ratios than xKV and CommonKV.
What the evidence shows
- Start with the memory bottleneck
BellowsKV is a training-free method for compressing a model’s key-value (KV) cache, the records each layer keeps for past tokens. Each request holds a separate cache for every layer, so longer inputs and larger serving batches increase memory use.
The research question is whether adjacent layers can share stored information and still recover their own attention keys and values.
- Find what layers share
The authors compared neighboring layers in uncompressed Llama and Qwen models, token by token. They used cosine similarity, a measure of whether vectors point in similar directions, for hidden states and cache entries.
Hidden states were strongly similar, while corresponding keys and values were close to zero similarity. That points to shared information in the inputs used to form the cache, rather than in cache entries that one layer could reuse.
- Reconstruct each layer’s cache
BellowsKV fits a projection from calibration data for each block of adjacent layers. It uses that projection to store one shared representation per token, then each layer applies its own projection to rebuild its keys and values.
The method keeps the shared representation at full dimension and compresses it either by storing values with fewer bits or by removing selected tokens using one decision per block.
- Measure retained task performance
LongBench consists of 11 English tasks involving question answering, summarization and finding information in long text. The xKV authors’ harness scored three instruction models on inputs longer than 4,000 and within 32,000 tokens.
The paper divided each compressed task score by the same model’s dense-cache score, then averaged across tasks. BellowsKV’s quantized version retained 97–98% at 13–14x compression; xKV retained 54–84% near 16x.
- Probe the rank explanation
A control compared xKV’s one-quarter-rank, fp16 basis at 16x compression with xKV-hold, which keeps that basis near full rank but stores it at four-bit precision and reaches 15.06x. Retention rose on Llama and Mistral, from 83.5% to 97.6% and from 79.5% to 99.2%, but fell from 54.4% to 23.4% on Qwen.
Since rank and precision changed together, the control cannot isolate either factor; the authors also have no explanation for Qwen.
- Check retrieval on other models
BrowseComp-Plus tests one-turn retrieval followed by answering, and its judge scores outputs on 830 queries. With a 32,768-token prompt filled with difficult distractor passages, BellowsKV’s scores at 14.61x compression were close to the dense run: 86.02 versus 85.54 on Qwen3-32B and 85.78 versus 86.14 on Qwen3-30B-A3B.
This adds evidence on two models and one task, rather than establishing performance across tasks generally.
Deep
- Original paper authors
- Jiamu Zhang, Hanjie Chen
- Submitted to arXiv
- License
- CC BY 4.0