HyScale speeds up million-token training for sparse attention
HyScale: Scaling Long-Context Training for Learned Sparse Attention · arXiv:2610.32057
In plain words
Some AI systems choose only a few earlier words to pay attention to, but finding those words can itself take a lot of time when the text is very long.
HyScale is a training system that finds the chosen words as it scores them and saves those choices so it does not have to find them again during training.
In the tested setup, training GLM-5.1 on one-million-token sequences was 2.07 times faster with HyScale than with Megatron-LM. That result applies to the reported model and 512 H20 GPUs.
The research
When an attention system uses only a few entries from a very long sequence, why can finding those entries still consume so much training time? In the measured GLM-5.1 setup, selection took nearly half of a training step, creating a specific opportunity to reduce the cost of long-context training.
HyScale combines on-the-fly selection and replay during recomputation to reduce the cost of training learned sparse attention on long sequences.
HyScale combines two changes to selection: it chooses entries as it scores them, then reuses the chosen indices during recomputation. This avoids writing a full score table to GPU memory and repeating the full selection scan. The system also includes optimizations for long-context training.
In a comparison of full GLM-5.1 training from released weights, HyScale is 2.07 times faster than Megatron-LM on one-million-token sequences using 512 H20 GPUs.
A separate layer ablation attributes most of its measured layer-time saving to the two selection methods, supporting the conclusion that reducing selection work contributes substantially in the tested settings.
These results support faster training and layer execution in the reported configurations. They do not establish the same speedups or training quality for other models, sequence lengths, or hardware.
What the evidence shows
- Selection is the bottleneck
HyScale is a long-context training system for learned sparse attention, which selects a few earlier entries for each query. The question is whether training can avoid spending more time selecting entries than using them.
In the measured Megatron-LM setup, selection took 47% of a GLM-5.1 training step at one million tokens, excluding pipeline wait, and lasted 6.6 times as long as sparse attention.
- Choose entries while scoring
HyScale’s HyTopK method estimates a threshold from evenly spaced keys in each query’s context. It filters scores as it computes them, avoiding a full score table in GPU memory, then selects the exact top-k entries from its candidates.
Because the estimate can fail, the system detects queries with too few candidates or an overflow and rescans their full context on the GPU.
- Reuse selections in training
Selection replay addresses a second cost: activation recomputation can repeat the full selection scan. HyScale saves the selected indices during the forward pass and uses them to rescore only those entries during recomputation.
This fits the backward pass because it needs the selected entries and their scores, not a new search across the whole context.
- Layer gains reach full training
In a cumulative ablation on one GLM-5.1 attention layer at one million tokens, HyTopK and selection replay account for 84% of the measured 66% reduction in Megatron-LM layer time; implementation optimizations account for the other 11% of baseline layer time.
In full GLM-5.1 training on 512 H20 GPUs, HyScale is 2.07 times faster at one million tokens. The layer ablation and end-to-end comparison measure different scopes.
- Two more layer tests
The authors also compare individual attention layers for two block-level architectures with their Megatron-LM implementations. At one million tokens on 32 H800 GPUs, HyScale runs the DeepSeek-V4-Flash CSA layer 2.01 times faster and the Qwen3.8-Flash-Next QSA layer 1.57 times faster.
These layer results show measured gains in those tests, not end-to-end training speedups for the two models.
Deep
- Original paper authors
- Yangsheng Deng, Xinjing Huang, Yulong Wang, Yunzhuo Liu, Jun Hu
- Submitted to arXiv
- License
- arXiv non-exclusive