Robot history helps most when video learning predicts forward

When a robot sees a moving object, how much past visual information helps it choose the next action, and can that history improve control without making responses too slow?

The open issue is whether longer context helps simply because more images are available, or because the model learned how earlier scenes predict what happens next.

Long-WAM trains a robot video model to predict from past scenes, then uses those predicted futures alongside visual history to generate actions.

The authors first train video prediction on robot and egocentric footage without action labels, then adapt the model so a video expert predicts future scene representations and an action expert uses those predictions with observed history.

On RoboCasa GR-1, a simulation test of multi-stage object transfers, separately trained models completed more tasks with 19.2 than 2.4 seconds of history, while keeping the visual forecast and action horizon fixed.

Yet the result depends on what the video model learned: with bidirectional video pretraining, success at 19.2 seconds was essentially unchanged from 2.4 seconds, while both autoregressive initializations improved.

This supports the authors’ interpretation that predictive use of history matters, not just access to it.

Each history length used its own trained model, and 80.4% of sampled history frames in the 38.4-second condition were padding. The context-length comparison was in simulation; the paper does not report a real-robot comparison across history lengths.