LayerRoPE turns layer depth into a signal for Transformer blocks

As information passes through a Transformer, hidden-state magnitudes can rise sharply with depth. Does this growth simply need to be suppressed, or can it carry useful information about a layer’s position? The answer matters for how a model’s blocks are regulated as networks grow deeper.

LayerRoPE generates each Transformer layer’s normalization gains from shared gains adjusted for depth, regulating block inputs and outputs.

To assess whether rising hidden-state magnitudes with depth are only a problem, the authors examined normalization gains in pretrained language models.

In most tested models, MLP-block gains varied in magnitude and direction with depth, consistent with an implicit layer-position signal, though exceptions appeared.

They then tested LayerRoPE, which generates depth-specific gains from shared gains, against other normalization methods in C4-trained language models. It had the lowest validation loss at each tested size from 58 million to 1.3 billion parameters.

A fitted curve estimated 3.4 times less compute than Pre-Norm to reach its 1.3-billion-parameter loss, supporting an advantage in these comparisons.

The strongest language-model results come from the paper’s tested C4 training settings. The compute saving is estimated from fitted curves, and the 512-layer fixed-rate result uses one seed; the pretrained-gain pattern also has model and block-specific exceptions.