Research collection
#normalization-gains
Research tagged normalization-gains: explanations, evidence, limitations and connections between papers.
1 results
LayerRoPE turns layer depth into a signal for Transformer blocks
LayerRoPE generates Transformer normalization gains from shared, depth-adjusted values and had the lowest C4 validation loss at tested scales.