A planner finds where streaming models should fuse GPU work
One Megakernel is Not Enough: Predicting Where to Cut for Streaming Inference · arXiv:2610.35336
In plain words
Streaming models produce output in stages, and different stages can work best with different GPU settings. Combining stages into one persistent kernel can save launch overhead, but making one giant kernel can also be a poor fit.
The paper introduces a planner that uses existing measurements and a model of when stages can run and publish output to choose where to combine them.
On one GPU, it picked the fastest measured design in all 13 tested model and batch-size cases, within the designs the authors had implemented and checked.
The research
When a streaming model produces its first output, how should its phases be grouped on the GPU? Different phases can run at different rates and prefer different launch configurations, and publication may depend on work from more than one phase. The best fusion boundaries are therefore not obvious from the model’s stages alone.
The planner combines existing profiles, compatible kernel measurements and a simulation of output dependencies to rank supported fusion designs without timing each one end to end.
The planner combines existing profiles, compatible kernel measurements and a simulation of output dependencies to rank supported fusion designs without timing each one end to end.
On an H100 NVL GPU, its frozen choice matched the fastest measured design in all 13 tested model and batch-size cases, comparing within each model’s implemented and qualified designs. The result supports the ranking method within those validation pools.
The 13-of-13 selection result applies to the designs the authors implemented and qualified on an H100 NVL GPU. It does not establish the best choice among unmeasured designs or on other GPU architectures.
What the evidence shows
- Fusion is a boundary choice
The first-publication-aware boundary planner is a method for selecting which contiguous regions of a streaming model to fuse into megakernels, which to run conventionally, and how regions share the GPU. The question matters because phases can run at different rates and prefer different launch configurations, while first output may wait on more than one phase. A boundary choice affects both GPU resource use and when output can be published.
- Rank supported compositions
Instead of timing every full candidate design, the planner combines conventional-execution profiles, operator or step measurements, and implementation details declared in source. It uses compatible measurements to estimate service costs, builds supported candidate compositions, and simulates how their work and publication dependencies proceed.
Candidates without compatible cost evidence are left out, so the result ranks only designs the planner can support.
- A measured case favors three
The NH-TTS comparison at batch size 32 shows why both fusion extent and overlap matter. On an H100 NVL GPU, a three-megakernel design with overlapping work published first audio 4.05 times sooner than conventional TensorRT execution and 21.4% sooner than a single megakernel.
These measurements show that, in this case, neither conventional execution nor fusing the entire model was the fastest tested design.
- Selections match measured winners
Across three models and three frameworks, the authors tested 13 model and batch-size cases on an H100 NVL GPU. In each case, the planner’s frozen choice matched the fastest measured design in that model’s implemented and qualified validation pool.
That supports its ability to select among those measured candidates; it does not show that those pools included every possible composition.
- Overlap and batch size matter
Two controls used the same service costs and candidate pools but replaced event simulation with additive scoring or removed overlap. Each matched 11 of 13 measured winners, missing NH-TTS at batches 32 and 64, where it chose serial execution.
Separately, Qwen3-TTS favored four megakernels through batch 32 but conventional execution at batches 48 and 64. These results support the view that overlap and useful fusion extent depend on the case.
Deep
- Original paper authors
- Aditya Chatterjee, Anand Joseph X M, Sanjay Singh Chauhan
- Submitted to arXiv
- License
- CC BY 4.0