A method finds feature changes that recur across prompts
Representation Redistribution in Sparse Autoencoders · arXiv:2610.21240
In plain words
A prompt can be changed by adding context, changing its format, or editing its wording. A feature that changes once may not change again, and even a repeated change may not affect a later decision.
The paper introduces a way to pick features that change repeatedly in the same direction, then checks them on different prompt variants. In two fixed safety calculations, those features helped rank cases where the calculation repeatedly made a wrong decision. The test was of the calculations, not of answers produced by the language models.
The research
When a prompt changes, a language model’s internal features may change too. But which changes recur across related versions, and do they matter to a later decision?
Answering both questions requires checking whether a pattern lasts beyond the examples used to identify it, then testing its connection to a specific downstream calculation.
RSS selects sparse features that change repeatedly in one direction, then tests whether those fixed changes persist and matter to frozen readouts.
Researchers asked whether feature changes under related prompt transformations recur beyond the examples used to identify them, and whether those changes affect a later decision.
RSS selects features by response size, recurrence, and directional consistency, then keeps that selection fixed for tests on separate variants.
Across two frozen safety readouts, selected changes persisted more than magnitude-only controls and helped rank cases with repeated readout errors when combined with the full clean margin. This links recurring representation changes to readout behavior while keeping the two measurements distinct.
The functional results come from offline tests of two frozen linear readouts, using new examples of the same prompt transformations. They do not show how generated answers behave or establish transfer to other transformations or changing readouts.
What the evidence shows
- When changes recur
A prompt’s wording and surrounding material can change which sparse autoencoder features respond. But one observed change may be specific to that prompt, and recurrence alone does not show whether a later calculation uses the changing feature.
The redistribution sensitivity score (RSS) is a method for selecting feature changes that recur in a consistent direction, then checking whether they persist on variants excluded from selection.
- How selection works
RSS first ranks features by the size of their average absolute change. It then keeps only features that respond across enough selection examples and do so mostly in one direction.
The selected features and directions are held fixed for evaluation on separate variants. This makes the test about whether a pattern carries beyond the examples used to find it.
- Patterns beyond selection
Across 1,000 English prompts and 27 pretrained autoencoders, context and wrapper transformations had higher relative response profiles than local edits to the preserved request. On excluded variants, RSS-selected features followed their selected direction more often than size-matched random features.
These results support a recurring pattern in the tested settings, rather than a claim that all prompt changes behave alike.
- From features to readouts
The researchers also tested two frozen safety readouts, which turn feature values into scores for decisions. RSS features retained their selected direction more often on new variants than features chosen by magnitude alone at the same support size.
Combining the full clean margin with erosion estimated from RSS features also ranked recurrent readout failures well. The findings connect recurring feature changes to these readouts, while leaving model-generated behavior untested.
Deep
- Original paper authors
- Ziye Mei, Haotian Ma
- Submitted to arXiv
- License
- arXiv non-exclusive