FP4 cache decoding keeps accuracy close while raising serving capacity
BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design · arXiv:2611.09825
In plain words
Language models keep a cache of information from earlier text so they can generate the next token. Storing it in a smaller four-bit format saves memory, but can distort information or slow down decoding.
BitDecoding-FP4 combines a way to store that information with GPU operations designed for the format. On two Qwen3 models, its average accuracy across seven long-context and reasoning tasks stayed within half a percentage point of a higher-precision cache. The result applies to those models and tasks.
The research
Longer contexts put pressure on the memory used to keep earlier text available, but compressing that memory can hurt answer quality. These results suggest that, on the tested Qwen3 tasks and serving workloads, a four-bit cache can preserve average accuracy while supporting higher peak request rates.
BitDecoding-FP4 pairs an FP4 cache quantization recipe with GPU kernels to keep accuracy close to BF16 while increasing tested serving throughput.
The authors compared BitDecoding-NVFP4 with a BF16 cache on seven long-context and reasoning tasks, measuring answer accuracy and averaging across tasks separately for each Qwen3 model. The FP4 averages were 67.27% versus 67.69% on Qwen3-4B and 72.74% versus 73.15% on Qwen3-30B-A3B.
Because both averages differ by less than half a percentage point, the results support similar average accuracy on these evaluations. In separate B300 serving tests, peak requests per GPU were 2.4 to 3.0 times the BF16-cache result for the stated contexts and concurrency levels.
The accuracy and serving comparisons cover the listed models, tasks, contexts, and hardware. The Hy4-preview result covers three tasks at a 1M-token cache length; the paper reports no latency comparison for that sparse-attention setup.
What the evidence shows
- A joint recipe and system
BitDecoding-FP4 is a decoding method and system that pairs a quantization recipe with GPU kernels for native FP4 attention. It addresses a trade-off in long-context generation: direct four-bit caches can lose accuracy, while some accuracy-focused approaches add work during decoding.
The design changes both how queries, keys, and probabilities are represented and how the GPU processes them.
- Accuracy stays close on Qwen3
For the main accuracy comparison, the authors averaged answer accuracy across seven long-context and reasoning tasks separately for each Qwen3 model. The averages were close to the BF16 cache results on both models.
These results support a close match on the listed evaluations, while leaving room for larger differences on individual tasks.
- A second query term cuts error
The query representation comparison helps explain one design choice. With NVFP4 keys held fixed, the query accounted for 60% of total mean squared logit error when stored as NVFP4, compared with 1.2% when represented by two FP4 terms.
This measures a component of attention-score error on one model, not end-task accuracy; it shows that the second term substantially reduced the query's share in that comparison.
- A larger model extends the result
A separate test extends the accuracy comparison to Hy4-preview, a 770B-parameter model using sparse attention. At a 1M-token cache length, BitDecoding-NVFP4 remained within 0.12 accuracy points of BF16 across three reported tasks.
That result supports accuracy transfer to this model and cache setting, but the paper gives no latency comparison for Hy4-preview.
- Serving throughput rises in tested workloads
Across the tested Qwen3 workloads, the FP4 cache delivered higher peak per-GPU request rates than the BF16 cache. This peak comparison covers B300 serving at 64K and 128K context, with concurrency varied from 1 to 64 requests; it is specific to those workloads.
Deep
- Original paper authors
- Dayou Du, Woon Yee Ng, Guangming Sheng, Nick Lau, Jianyi Cheng
- Submitted to arXiv
- License
- Not specified by source