Video generators can stop refining tokens that have settled
In plain words
The paper introduces Speculative Forcing, a way for video generators that build videos piece by piece to spend less time refining parts that already seem settled. It checks each small part of the video: settled estimates are kept, while the rest continue to be refined. Kept parts still help the generator make what comes next.
On five-second LongLive videos, the learned check used fewer than half as many denoising evaluations per token as the usual four-step process, while the aggregate score on a video test stayed close.
The research
If some parts of a generated video stop changing early while others still need work, does the generator have to spend the same number of refinement steps on every part? The paper asks whether it can save computation by deciding token by token when an estimate is ready to keep.
Speculative Forcing lets a few-step video generator freeze token estimates predicted to have settled, so later denoising work focuses on the rest.
The method checks intermediate token estimates and freezes some early, while the rest continue through the same four-step schedule. On LongLive’s five-second videos, its learned verifier averaged 1.68 denoising evaluations per token, versus 4.00 for uniform four-step sampling.
VBench, a test that rates video quality and prompt-related meaning, gave aggregate scores of 83.21 and 83.08, respectively. The close scores support comparable aggregate performance in this evaluation despite the reduced denoising work.
These results cover two frozen generators in the tested settings. The authors say attainable acceleration depends on the generator and the difficulty of the domain, so the same speed gain is not established elsewhere.
Analysis
Deep
- Original paper authors
- Hyunwook Choi, Seungryong Kim, Dongyoon Han, Sangdoo Yun
- Submitted to arXiv
- License
- CC BY 4.0