Training can shrink the weights most vulnerable to bit flips
LEAN: Loss-Engineered Modulation of the Model Weight-Sensitivity Distribution for Efficient, Error-Tolerant Neural Networks · arXiv:2610.28872
In plain words
Some stored weights in a neural network matter much more than others when a bit flips. The paper introduces LEAN, a training method that reshapes this unevenness by reducing the sensitivity of the most vulnerable weights and the size of the flip step in each layer.
Across nine vision model and dataset combinations, LEAN reduced the count of highly sensitive weights by 5 to 13 times, while clean accuracy changed by no more than half a percentage point. These results describe the tested models and training runs.
The research
When a stored bit flips, damage depends on which weight it changes: a small group can matter far more than the rest.
Can training reduce that concentration, so an attacker has fewer especially damaging weights to target and selective protection has less to cover, while preserving ordinary model accuracy?
LEAN reshapes stored-weight sensitivity during training, shrinking the highly sensitive tail across tested vision models; in one ResNet-18 CIFAR-10 check, real bit flips caused less measured damage.
LEAN adds a range penalty and a penalty on the most sensitive weights while training a model further. Researchers compared baseline checkpoints with models trained for ten extra epochs, measuring sensitivity on held-out batches against thresholds set from each baseline. Across nine vision configurations, the HIGH group shrank while clean accuracy changed little.
To check that the score reflected real damage, researchers flipped stored sign bits in one ResNet-18 CIFAR-10 model and measured the loss increase. It fell for the worst weights with both LEAN terms.
With a 1% protection code, the LEAN model also withstood more progressive flips than its baseline in the reported test.
These results come mainly from vision models and specific bit-flip procedures. The code tests assume interleaving prevents too many errors from accumulating in one codeword, and the code itself was not simulated. Most reported entries use one training run and one attack run.
What the evidence shows
- A concentrated source of damage
LEAN is a training method that reshapes how much stored weights matter when their bits flip. The question is whether training can reduce the concentration of damage in a small group of weights, since adaptive attacks can target that group and selective protection schemes need to cover it.
In the paper’s ResNet-18 CIFAR-10 example, the most sensitive 1% accounted for about 20% of the total first-order damage from sign-bit flips.
- Two parts of the training change
The method penalizes two contributors to flip damage: the gradient of a weight and the step a bit flip makes in the storage format. Its range term limits each layer’s INT8 range, reducing the flip step; its tail term penalizes the mean sensitivity of the most damaging 1% of weights. A coefficient sets the strength of the combined penalty.
- The sensitive tail shrinks
Researchers compared each baseline model with a LEAN model trained for ten more epochs from the baseline checkpoint. Across nine ResNet and VGG configurations on CIFAR-10, CIFAR-100 and ImageNet, LEAN reduced the count of weights above a threshold fixed from the baseline by 5 to 13 times.
Clean accuracy changed by at most 0.5 percentage points, and the top 1%’s share of total first-order damage fell in every configuration.
- Real flips test the measure
A score alone cannot show that actual flips became less damaging, so researchers tested real INT8 sign-bit flips in ResNet-18 on CIFAR-10. Mean loss increase among the 100 worst weights fell from 0.031 for the baseline to 0.009 with both LEAN terms.
In this setting, the range term did most of the work; the tail term added to it, while the tail term alone could lower the score without lowering actual damage.
- Protection improves in tested cases
With a 1% error-correcting code, the baseline ResNet-18 on CIFAR-10 fell below 90% accuracy by 200 progressive bit-search flips. LEAN retained 94% through 300 flips in all three seeds and through 500 in two of three.
A separate test found that LEAN needed less code parity than the baseline to meet one specified protection guarantee.
Deep
- Original paper authors
- Habibur Rahaman, Swarup Bhunia
- Submitted to arXiv
- License
- CC BY 4.0