ReadKV keeps cache detail without reading it all
In plain words
ReadKV is a method for compressing the saved key, value information a language model consults while continuing a long text. It keeps a more detailed stored version, but does not read every stored bit at every step.
For each new query, it first reads the key detail that matters most, estimates which earlier tokens matter, and then reads more value detail there.
On six language models, an eight-bit stored cache read at four bits on average stayed within 0.66% of an uncompressed cache’s perplexity, a next-token prediction score, while using about one quarter of its counted reads.
The research
When a language model handles a long context, it repeatedly consults saved information from earlier tokens. It remains unclear whether retaining extra detail for future queries can reduce each read without simply throwing away information.
That matters here because stored capacity and per-step read volume are separate costs in the paper’s cache model.
ReadKV separates stored precision from fetched precision, letting each decoding query read only the key and value detail it needs.
ReadKV stores each key and value scalar in a progressive code: longer prefixes reconstruct it more precisely. For a new query, it allocates key prefixes from the query, computes attention from those reconstructed keys, and allocates value prefixes from the resulting attention. This separates retained detail from the average payload read per scalar.
Across six C4 document tests of next-token prediction, ReadKV (8,4) increased perplexity, where lower is better, by at most 0.66% versus Dense, the uncompressed 16-bit cache, while using about 25% of its logical reads and 50% of its retained capacity.
In six 7,500-token question-answering conditions, (8,2) used about 13% of Dense logical reads, with token-F1 changes from −0.66 to +1.46 points. The evidence supports a quality–read tradeoff, not a uniform win over every comparator.
The theoretical separations apply only to specified transformed-sparse or specially constructed attention families, and the allocation theorem is stagewise rather than a joint guarantee for true attention-output error.
The GPU result is a single-layer, batch-one A10G measurement from an existing cache, so it does not establish end-to-end serving or throughput gains.
Analysis
Deep
- License
- arXiv non-exclusive