FOSCAR speeds attention by folding quantization into matrix products
FOSCAR: Accelerating Decode Attention over 2- and 4-Bit KV Caches through Affine Folding · arXiv:2611.09722
In plain words
When a language model generates text, it repeatedly reads a stored cache of keys and values. Storing that cache in fewer bits saves memory traffic, but the GPU still has to rebuild each value before using it, which can eat into the time saved.
FOSCAR is a GPU method that moves shared scale and offset calculations into attention’s matrix products.
In timing tests on H100 and B200 GPUs, its two-bit kernel had lower latency than the BF16 comparison kernel, where BF16 is a 16-bit floating-point format, across the tested batch sizes and context lengths.
The research
Low-bit storage promises less data to move, but that alone does not guarantee faster token generation: the GPU can spend its saved time unpacking and rescaling values.
FOSCAR shows that this preparation can instead be handled inside attention’s matrix products, changing what a kernel comparison should measure when judging whether compressed caches actually reduce decode latency.
FOSCAR is a GPU kernel method that folds shared cache scales and offsets into attention’s matrix products, avoiding per-value reconstruction.
FOSCAR moves each shared scale and offset into another input or the partial result of attention’s matrix products, while bit manipulation builds operands from packed integers. In exact arithmetic, the output matches reconstructing each cache value first.
On H100 and B200, standalone timing tests found lower average latency for FOSCAR’s two-bit kernel than the FlashInfer BF16 kernel, where BF16 is a 16-bit floating-point format, across the tested batch and context grid.
The reported geometric-mean speedup combines case-by-case ratios into one multiplicative average, supporting a kernel-level benefit.
The kernel speedups summarize a tested H100 and B200 batch-and-context grid, so they do not specify the gain in every case. The serving comparison covers one Qwen3-8B long-context workload, and FOSCAR is designed for uniform affine cache quantization.
What the evidence shows
- Where low-bit savings go
FOSCAR is a GPU kernel method for decode attention, the step where a language model reads its stored keys and values to generate the next token. Fewer bits reduce cache traffic, but existing kernels still reconstruct each value and apply its scale and offset before multiplication. That work can use instruction capacity and limit how much the smaller cache speeds attention.
- Latency fell on both GPUs
Standalone kernel timing tests measure how long decode attention takes to run. A speedup compares the baseline latency with FOSCAR’s, and the geometric mean combines those ratios across tested cases.
Across the H100 batch-and-context grid, FOSCAR INT2’s geometric-mean speedup over FlashInfer BF16, a 16-bit floating-point baseline, was 3.36x; across the B200 grid, it was 2.67x. This indicates lower average kernel latency on both GPUs under the tested conditions.
- A test isolates folding
The overall comparison alone does not isolate the effect of folding. A separate B200 test compared an INT2 kernel that reconstructed each value into FP8 operands with one using affine folding, keeping the number of context splits equal.
Folding was 7.36–8.86x faster in that comparison. This supports the idea that removing per-value reconstruction contributes to the B200 kernel result, within that specific test.
- The gain reached serving
In this single-GPU Qwen3-8B workload, FOSCAR’s INT2 setup raised peak aggregate output throughput over OSCAR INT2 by 88% on H100 and 170% on B200. This comparison used 100K input tokens, 1K output tokens, and a 99% prefix-cache hit rate; peak throughput also reflects how many requests fit in memory.
Deep
- Original paper authors
- Donglin Zhuang, Pragaash Ponnusamy, Xiaoxia Wu, Shuaiwen Leon Song, Zhongzhu Zhou
- Submitted to arXiv
- License
- Not specified by source