Grouped residual updates cut training overhead while keeping quality competitive
On the Design of Hyper-Connections: Simplification, Stability, and Scalability · arXiv:2611.05984
In plain words
Some language models use several streams of information that layers can read from and update. SimpleHC is a design that keeps learned choices about those reads and updates but groups layers so they can share a starting state.
Small corrections carry earlier layer outputs forward, and updates are written back at the end of each group.
In the reported tests, SimpleHC’s average scores were close to other Hyper-Connections designs, and a separate timing test found less added training time at its largest tested width. These results come from different model setups.
The research
Richer ways to route information through a language model can add quality, but repeated reads and writes of its residual streams also add training time. The results suggest that a model can keep learned routing and comparable test scores with less measured overhead in the configurations studied.
SimpleHC groups Hyper-Connections layers around a shared starting state to reduce repeated residual-stream access while retaining learned read and write gates.
In a 30B-A2B model, the researchers compared SimpleHC with mHC and identityHC across fourteen tests covering knowledge, reasoning, mathematics, code, and Chinese. SimpleHC averaged 63.1, versus 62.9 and 62.8, respectively, though it did not lead on every test.
A separate eight-layer timing test found 2.4% added training time for SimpleHC at width 6144, compared with about 7.0% for each alternative. Together, these findings support comparable test performance with lower added time in the measured setup.
The results come from separate model sizes and training setups, and the timing comparison used eight-layer models. The execution-equivalence proof applies only when both procedures use gates generated from the same group-entry state.
What the evidence shows
- Separate useful components
SimpleHC keeps learned read/write gates and removes residual mixing. In the 15B-A1B comparison, learned gates and unconstrained mixing each performed comparably to mHC, while combining them scored higher: 56.2 versus 55.0 for mHC.
- Why remove residual mixing
The component comparison suggests that learned read/write gates and unconstrained mixing can each perform comparably to mHC, with a higher score when combined. But unconstrained mixing can amplify or weaken signals as they pass through layers, while the constrained version added little over learned gating alone.
The authors therefore selected learned gates without mixing as their minimal effective structure. This is a design conclusion from the studied model and tests, not a general result for every architecture.
- Corrections preserve contributions
SimpleHC shares the group-entry state used to generate layer-specific gates, but later layers still need earlier layers’ outputs. A test of this point found validation loss of 1.660 with scalar corrections and 1.705 without them on the Pile test split.
Both versions wrote all group outputs back at the boundary. The result supports the corrections’ role in preserving contributions to later inputs.
- Comparable scores at larger scale
Across fourteen tests on a 30B-A2B model, SimpleHC’s average was marginally higher than the alternatives, but individual results varied. It trailed mHC on a math word-problem test and both alternatives on code generation, so the average indicates comparable results rather than a consistent lead.
- Cost and training behavior
In an eight-layer timing comparison at width 6144, SimpleHC added less training time than mHC and identityHC. During standard 15B training, its largest 1,000-step rolling fluctuation in gradient norm after warmup was also smaller, though all three methods later reached comparable ranges.
Deep
- Original paper authors
- Xiangdong Zhang, Xiaohan Qin, Sunan Zou, Tuo Dai, Bizhao Shi
- Submitted to arXiv
- License
- arXiv non-exclusive