CanonQ trains language models to use one shared low-bit codec
In plain words
Language models store numbers for their weights, working activations, and attention cache. Compressing all three to very few bits is difficult because their numbers differ and compression errors can interact.
CanonQ transforms these values into shared coordinates, uses fixed sets of representative values, and trains the model while compression is active. In one LLaMA3-1B test with 2-bit weights, 4-bit activations, and a 2-bit cache, it had higher average accuracy than ParetoQ across eight question-answering and related tests. That result applies to the tested configurations.
The research
When a model compresses its weights, working activations, and attention cache at once, errors in one part can affect the others. It remains unclear whether one reusable quantization design can support all three while the model learns to operate through their combined errors.
CanonQ transforms model tensors into canonical coordinates, then trains the model through fixed quantizers to adapt to their interacting errors.
For LLaMA3-1B at 2-bit weights, 4-bit activations, and a 2-bit KV cache, the authors compared trained CanonQ and ParetoQ configurations using different quantization paths. On WikiText-2, a text corpus scored by next-token prediction error, perplexity was 15.42 for CanonQ and 90.57 for ParetoQ.
Across eight zero-shot question-answering and related tests, mean accuracy, the share of correct answers without examples in the prompt, was 53.56% for CanonQ and 40.47% for ParetoQ. Both measures favor CanonQ in this tested comparison.
The headline comparison is between complete trained configurations, so it does not isolate the quantization method as the only difference. Each configuration uses one training seed, and the findings cover the models and settings tested in the paper.