Binary models can train with small update buffers
Error Diffusion Replaces Latent Weights: Training Binary-Weight Language Models on a Few Bits of Dithered State · arXiv:2610.34566
In plain words
Binary-weight models store each weight as one of two signs, but the usual training method also keeps a full-precision copy and optimizer records for every weight. That extra state can be much larger than the model’s binary weights.
Error Diffusion is an optimizer that stores small leftover updates in a buffer until they are enough to flip a weight’s sign. In the paper’s synthetic network test, a 4-bit buffer came close to the standard method’s accuracy while using much less per-weight training state. Language-model tests also found similar losses in some settings.
The research
Binary models store weights compactly, but the usual training method keeps much more state for each weight than the deployed model does. Can a binary model keep learning without rebuilding that large training copy? The paper tests an optimizer designed to retain only small pieces of unrealized updates.
Error Diffusion replaces full-precision training copies with small update buffers, and tests whether binary models can learn with that smaller state.
The paper compares Error Diffusion with the standard latent-weight approach, which keeps a full-precision copy and optimizer moments for every binary weight.
In a width-192 synthetic MLP, ED’s 4-bit buffer averaged 56.4% test accuracy across five seeds, near the reference’s 57.5%, while using 5 rather than 97 bits of state per weight.
Language-model tests then found ED near the latent reference in several settings. With a per-step update cap, the 8-bit buffer’s validation loss ranged from 2.3% below to 2.9% above the reference across six tested model-size and corpus combinations; the separate 4-bit ladder had larger gaps and some wikitext-103 runs diverged.
Without the cap, a large tokenized-model run had a larger loss gap. This supports the method in the tested recipes, including the cap condition.
These results cover a synthetic MLP and language-model validation tests, with binary cores up to 19 million parameters. The longer Shakespeare and some tokenized-model comparisons use one run, and the study does not test downstream tasks or wall-clock efficiency.
What the evidence shows
- Replace the large training copy
Error Diffusion (ED) is an optimizer for training binary-weight neural networks. The usual recipe stores a full-precision latent weight and two optimizer moments alongside each one-bit weight, totaling 97 bits of persistent training state per weight.
ED asks whether training can instead carry unrealized updates in a smaller state while still learning useful weights.
- Compare accuracy and state
In a synthetic teacher–student multilayer perceptron (MLP), test accuracy measures how often the student classifies held-back examples correctly. At width 192, ED with a 4-bit buffer averaged 56.4% across five seeds, near the latent reference’s 57.5%; their per-weight states were 5 and 97 bits.
The evidence is specific to this synthetic test, but it shows the smaller state can approach the reference there.
- Test why grid placement matters
The buffer’s grid placement mattered on that same MLP. With a 4-bit buffer, aligning its grid to the weight-flip threshold raised mean test accuracy from 53.2% to 56.4% across five seeds.
A separate test of dithered Bop, which flips weights when a gradient average crosses a threshold, also improved when its low-bit grid was aligned. These comparisons support threshold placement as a useful design rule in the tested cases.
- Check longer language-model runs
On character-level Shakespeare, ED with an 8-bit buffer stayed within the paper’s 20% loss criterion against latent training at all three widths after 2,000 steps; at 50,000 steps, one width-256 run was 1.7% higher in validation loss.
On six larger-model and corpus combinations, ED with an 8-bit buffer and per-step updates capped at 0.25 flip units ranged from 2.3% below to 2.9% above the reference. The separate 4-bit ladder had larger gaps, and some wikitext-103 runs diverged. The cap condition matters: without it, a large wikitext-103 run had a 10.2% higher loss.
- Try continued training
Finally, the authors adapted a deployed binary model to Shakespeare bytes. After 5,000 steps, ED’s mean validation loss across three seeds was close to continued latent-weight training, which had to rebuild its full training state.
A separate one-seed CPU run stayed under a stated 275 MB additional-memory cap. These demonstrations show what happened in the tested setups, while leaving broader tasks and efficiency questions open.
Deep
- Original paper authors
- Pierre-Olivier Bonin
- Submitted to arXiv
- License
- arXiv non-exclusive