Cache translation can speed model handoffs
Translating KV Caches Between Hybrid-Attention Language Models: MLA Latents, Linear-Attention States and Sparse-Attention Indexers · arXiv:2611.12062
In plain words
When a conversation moves to another language model, the new model usually has to read the whole conversation again, or rely on a shorter summary that may leave things out. This paper introduces a method that translates the first model’s stored prompt information into the format the next model uses.
The method handles different kinds of stored information in different ways. In one tested model pair, adding token identities made the receiving model much better at recalling exact six-digit codes in long prompts. The tests cover a limited set of models that share a tokenizer.
The research
A model switch can make a conversation slower or leave details behind in a summary. Cache translation offers a way for the receiving model to continue from stored prompt information, and the tests show both when exact recall improved and why task scores alone may miss a behavioral change.
KV-cache translation converts one language model’s stored prompt information into another’s cache format, letting the receiver continue without rereading the conversation.
The method translates each model’s stored prompt objects into the target’s format, using token identity where compressed cache values alone miss exact details.
Tests found near-native task results in one hybrid-to-full-MLA direction, plus a 6-in-100 refusal difference in another pair that KL and accuracy missed.
At 64,000 tokens, one measured cache switch reached the first generated token 18.7 times sooner than rereading. Together, these findings support cache translation in the tested settings while showing that task metrics and timing alone do not capture every behavioral difference.
The results cover tokenizer-sharing model pairs, with some full-set comparisons and the sparse-attention rehearsal using one seed. The tests do not establish that the method will work at flagship scale or across different tokenizer families.
What the evidence shows
- A different kind of handoff
KV-cache translation is a method for handing a conversation from one language model to another by converting the source’s stored prompt information into the target’s cache format. The target can then continue without rereading the prompt.
This addresses a tradeoff: rereading takes time, while a text summary can omit details. The tested models share a tokenizer, and the method trains a translator while keeping both models frozen.
- Identity restores exact recall
In Pair A’s full-MLA-to-hybrid direction, adding token identity raised exact six-digit-code recall in long prompts. This shows that the translator needed token identity to preserve exact details in this tested direction; mapping compressed cache values alone did not reliably retain them.
- Transfer can be fast
The reverse Pair A direction, from a hybrid model to a full-MLA target, reached results within paired confidence intervals of the target’s native-cache results on every reported task metric. This supports transfer for those tested tasks, although the comparison used one full-set seed and some point estimates were below the native results.
The direction also had a measured timing advantage: at 64,000 tokens, switching reached the first generated token 18.7 times sooner than rereading.
- A behavioral gap remained
Task scores did not capture every behavioral difference. In Pair B’s reverse direction, the translated cache led the target to refuse 6 of 100 needle prompts as sensitive, compared with no refusals from its native cache.
The paper’s KL and accuracy measures did not detect this difference. Cache transfer therefore needs behavioral checks alongside task scores in the tested setting.
- Sparse transfer has limits
The method also supplied predicted selection keys to one sparse-attention target. On the target’s own cache, predicted keys kept exact needle recall at 91 of 92 items, matching native keys.
With a translated cache after 400 steps, mapped Gated DeltaNet (GDN) state gave 0.86 recall across 92 needles, while zero state gave 0.13 despite similar validation KL (0.063 versus 0.078). Continuing the mapped-state run from step 400 under a 1,000-step schedule yielded 0.84 recall across 112 prompts from 2,000 to 64,000 tokens, versus 0.85 with native keys and 0.91 on the target’s own cache.
Recall was lower at 4,000 and 8,000 tokens. This one-seed rehearsal on one cross-model pair does not establish results for other sparse-attention systems.
Deep
- Original paper authors
- Povilas Kvedaras, Tomas Rasymas
- Submitted to arXiv
- License
- CC BY 4.0