A finer weight grid brings multiplier-free LLM inference closer

Can language models use weight values that make arithmetic simpler without giving up much accuracy? Standard power-of-two quantization offers shifts and additions instead of integer multipliers, but its widely spaced levels are an accuracy obstacle at four bits.

FPoT uses a finer logarithmic grid for weights, pairing it with integer activations to support multiplier-free shift-and-add inference.

The authors ask whether a finer weight grid can reduce the accuracy cost of using power-of-two values for multiplier-free language-model inference. FPoT places weight levels halfway between standard power-of-two levels and keeps activations uniformly quantized as integers.

On Llama-3-8B at W4A4KV4, FPoT's WikiText-2 perplexity is 7.60 versus 7.36 for uniform integer weights; its average accuracy across nine zero-shot tests is 64.8% versus 65.2%.

Across seven models and 21 settings, that accuracy average is within one percentage point of uniform integers in 20 settings, showing close results in most reported cases.

The model results use a particular preprocessing and calibration pipeline, and the detailed Llama-3-8B comparison reports three calibration seeds. The hardware figures are estimates from a simplified component-cost model; the paper leaves end-to-end chip validation for future work.