In plain words
Large language models can be made smaller by using fewer bits for each value or by dropping some values in a fixed pattern. This paper measures which choice better preserves the model’s calculations when storage or computing speed is matched.
Across two 8B models, using fewer bits while keeping all values usually caused less error than dropping values. At the GPUs’ lowest dense precision, however, a retrained sparse model served faster in one H100 setup, with no significant difference on a test of math problems.
The research
When compressing a large language model for inference, should the next step be to drop values in a hardware-required pattern or lower the precision of every value? The answer could change at a GPU’s lowest supported dense precision, where there is no lower-precision option left to compare.
The study matches 2:4 sparsity against lower precision by storage and compute, finding dense formats more accurate above the GPU’s lowest supported precision.
Researchers compared dropping values in hardware-required groups with lowering precision while keeping all values. They measured layer-operation error across six operand families in two 8B models, matching storage budgets (including sparse indices) and nominal compute tiers.
Dense quantization had lower error at every tested equal storage budget, and dense FP8 or NVFP4 had lower error than 2:4 at matched compute tiers. The authors link this pattern to energy lost from discarded values.
At the FP8 floor, one H100 serving comparison found higher prefill throughput for a retrained sparse model, without a significant GSM8K accuracy difference.
These results cover the tested formats, two 8B models, GPUs, and procedures. The Blackwell retraining comparison used one untuned run per student for about 1B tokens, so it does not show whether a longer or differently tuned run would close the accuracy gap.