Noisy future video can guide robot actions in one pass
What Makes World Action Models Generalize? An Empirical Study of Test-Time Future Modeling · arXiv:2609.34981
In plain words
A world action model is a robot controller that uses video to represent what may happen next while choosing movements. This paper compares controllers that give future video to their action predictor with controllers that leave it out at inference.
On the tested tasks, dropping future video kept success close on familiar tasks but did worse under scene changes and on held-out tasks. Simple-WAM keeps future video as noise, reads it once, and scores higher than both approaches on simulated environmental changes.
The research
When a robot controller learns to predict future video, must it generate clean future frames while choosing an action? Earlier latent models matched explicit models on familiar tasks with lower inference cost, but that did not show whether they retained the generalization linked to future modeling.
The authors compare future-video conditioning in robot action models and introduce Simple-WAM, which keeps noisy future video for a single pass.
In matched configurations, both models predicted future video during training. At inference, the explicit model progressively denoised it while the latent model dropped it.
They had similar success on familiar tasks, but explicit conditioning did better under scene changes, fewer demonstrations, and held-out tasks with video that had no action labels. A fixed-model test found one pass over noisy future video recovered nearly all the gain, suggesting clean frames were unnecessary in these tested settings.
These comparisons cover the tested simulated tasks and one held-out task on a physical robot; they do not establish the same results for other tasks or settings. The authors leave open whether the findings extend to larger models or models with large-scale embodied pretraining.
What the evidence shows
- The question about future video
The paper is a controlled empirical study of world action models, robot controllers that pair a video expert, which models future frames, with an action expert, which predicts movements. It asks whether a controller must generate clean future video while selecting actions.
Both explicit and latent models keep future-video prediction in training; they differ in whether future video guides actions at inference.
- Where the models diverge
In matched configurations, the authors measure task success: how often a robot completes a task. Explicit and latent models are nearly tied on familiar tasks.
But the explicit model scores higher under environmental changes, with only ten demonstrations per task, and on held-out tasks. The gap is largest when video of a held-out task without action labels is available to train the video branch.
- Testing the role of denoising
To test whether clean video is necessary, the authors vary video denoising at inference for the same explicitly trained model, without changing its weights or action denoising. On a simulated spatial-manipulation test, almost all measured gain is present after the first pass over a fully noisy future; later video passes add little.
The authors interpret this as evidence that the action expert benefits from reading future video, not from clean generated frames.
- A one-pass method
Simple-WAM is the authors’ method: it keeps future video as noise for one video-expert pass and trains more often at that noise-level while retaining the rest of the training schedule. Without large-scale embodied pretraining, its task-success score across seven environmental changes exceeds both explicit and latent configurations.
On one RTX 5090 GPU, producing an action chunk takes less time than with the explicit model and slightly more than with the latent model.
- Transfer to a physical task
On a physical dual-arm robot, the authors test the held-out Store in Order task after training on four other tasks. The reported score is the mean fraction of listed sub-goals completed per trial, averaged across 30 trials, rather than an all-or-nothing success rate.
It rises from 2.2% without held-out-task video to 87.8% when video without action labels trains the video branch.