In plain words
A draft model helps a larger model generate text by suggesting tokens. To save memory, the draft can keep only part of its record of earlier tokens, but the useful parts may change as it writes.
RIFT is a family of ways to keep both recent tokens and selected prompt context, and to refresh that selection during generation. In one Qwen test, sharing the cache between recent and selected context improved average accepted tokens over either a fixed prompt selection or a recent-token-only cache. The finding is limited to that tested setting.
The research
When a draft model has a fixed memory budget, which earlier tokens should it keep as it generates? A prompt selection can preserve distant context, while recent tokens capture what has just been written.
The paper examines whether the balance between them changes during generation and tests ways to adapt the cache.
RIFT splits a draft model’s fixed cache between recent tokens and selected prompt context, then can refresh selections as generation proceeds.
The paper compares two ways to spend a limited draft cache: keep prompt positions selected before generation, or keep only recent tokens. In one Qwen setting, their average acceptance patterns moved in opposite directions as generation progressed.
A full-cache attention measure also shifted toward the positions kept by the recent-token method, consistent with changing context relevance.
RIFT combines recent tokens with selected prompt context, and some variants refresh selections during generation. In the tested settings, this combination and refreshing improved average acceptance measures over the corresponding fixed-selection approach.
The evidence supports these methods for the reported models and conditions, rather than a universal advantage.
These results measure draft acceptance under particular models, cache budgets, and generation settings. The staleness pattern differs by architecture, and refreshing selections requires keeping the full draft cache history in memory or offloading it.
Analysis
Deep
- Original paper authors
- Dor Tsur, Maor Ashkenazi, Maximilian Golub, Peter Hicks, Yaniv Galron
- Submitted to arXiv
- License
- CC BY 4.0