Frozen-model output shifts match many weight-based gains
Rethinking Neural Thickets Without Weight Perturbations · arXiv:2610.40649
In plain words
The paper asks whether a language model’s weights need to change to get better answers from a group of varied versions. Its method keeps the model frozen, nudges the scores it assigns to possible next words, checks those versions on labeled examples, and votes with the best ones.
Across the authors’ original tests, this approach was statistically indistinguishable from changing all the weights in 22 of 28 model-and-task pairs. The result covers those models and tasks, not every model or use.
The research
Random weight changes have been reported to improve a pretrained model on specialized tasks, with the gains attributed to nearby task experts. Do those gains actually depend on changing weights, or can a method that varies only a model’s outputs reproduce them?
Perturb-Select-Vote tests whether validation-selected output diversity and voting can reproduce gains without changing a model’s weights.
The authors compare logit PSV with full-weight PSV under a matched version of the original protocol. Across 28 model–task pairs, paired 95% bootstrap intervals for their accuracy differences include zero in 22.
This means the test does not distinguish the methods in those pairs, while the remaining results favor each method in some cases.
Because the comparison retains selection and voting but changes the perturbation space, the pattern supports output diversity as an explanation for many of the observed gains. It does not establish that weight-space structure is absent or irrelevant in every setting.
The language comparison covers four models up to 7B parameters and seven tasks under the authors’ original protocol; it does not establish equivalence for every model or task. The method also needs labeled selection examples and multiple model evaluations per test input.
What the evidence shows
- Test the weight-space explanation
Perturb-Select-Vote (PSV) is an inference-time method that makes varied predictors, scores them on labeled examples, keeps the strongest, and combines their answers by voting. The paper asks whether gains previously attributed to nearby task-specialized models in weight space require changing model weights.
Its answer is tested by retaining PSV’s selection and voting steps while moving the source of variation from weights to outputs.
- A local change recovers gains
First, the authors perturb only one decoder block of OLMo-3-7B-Instruct on Countdown, an arithmetic puzzle task scored by the share of puzzles answered correctly. With 50 selected members, the greedy model scores 66.6% and full-weight PSV scores 90.8%; changing one block recovers much of the gain.
This diagnostic suggests whole-network changes are unnecessary in that setting, but does not yet show that changing outputs alone is sufficient.
- Outputs match in most pairs
The central comparison uses the authors’ original protocol across 28 model–task pairs: four language models and seven tasks. Both methods use 5,000 candidate members, select 50 using the same 200 training examples, and share the released prompts, splits, scorers, and generation limit.
Paired 95% bootstrap intervals include zero in 22 pairs; in the others, logit PSV has both advantages and disadvantages. The evidence supports similarity in most tested pairs, not universal equivalence.
- Selection depends on the case
The controls show why selection remains part of the account. Temperature-tuned majority voting, which samples and votes without validation-based selection, beats full-weight PSV on MATH-500 for OLMo-3-7B-Instruct.
But on Countdown with Qwen2.5-1.5B-Instruct, its best result across temperatures trails full-weight PSV. The authors read this contrast as evidence that voting alone can suffice in some strong-model cases, while selecting persistent candidates can help when samples are unreliable.
- The account extends with limits
In image classification with DINOv3-7B, output perturbations improve worst-region or worst-group accuracy on two shifted datasets, while backbone perturbations add little; balanced ImageNet shows no substantial gain. A deterministic correction shifts class scores toward validation-set class frequencies.
It recovers most of logit PSV’s gain on FMoW-OOD and matches its 5.0-point gain on binary Waterbirds, supporting a recalibration explanation under shift. A local mathematical check also finds that a first-order approximation reproduces many changed token choices at fixed prefixes.
Together these results support an output-based explanation in the tested settings, while leaving larger models and complete generated sequences unresolved.
Deep
- Original paper authors
- Nicolas Dufour, Elouan Gardès, Amil Dravid, Patrick Pérez, Loic Landrieu
- Submitted to arXiv
- License
- CC BY 4.0