EN
A robot model learns scene changes without predicting video
PDRVLA: A Vision-Language-Action Model with Physical Dynamics Representations · arXiv:2610.37381
In plain words
A robot handling objects needs to account for how the scene changes as it acts. PDRVLA is a model that turns camera images and instructions into robot commands, using a compact summary of predicted future appearance, shape, and movement.
The model learns that summary with future-scene examples and help from a video model during training, then acts without running that video model. In a test of future predictions, the added help lowered average error; separate robot-task tests were conducted in simulation.
The research
When a robot moves an object, the scene and its spatial relationships change. How can a robot use information about those changes when choosing actions without adding the latency of generating future video inside its control loop?
PDRVLA learns future-scene clues during training and uses them with current visual and robot-state information to generate actions without online video generation.
PDRVLA’s Physical Dynamics Module learns compact clues about future scene appearance, geometry, and motion. During training, a frozen video model supplies auxiliary guidance; the robot’s action expert later uses the learned tokens without running that video model during control.
In a held-out comparison, the assisted run had lower equally weighted prediction error across image, depth, and motion than direct-only training. Separately, PDRVLA reported strong task-success averages across four simulation tests, with a task-level exception.
The prediction comparison measures future-feature error, not action success, and does not establish that the added supervision caused higher success on the simulation tests. Policy evaluations cover four simulation settings; they do not establish performance with real robots.
What the evidence shows
- From scenes to actions
A robot manipulation model must connect what it sees now with how objects and their spatial relationships may change as it grasps, moves, and places them. Predicting future video can provide such information, but running video generation within the control loop adds latency.
PDRVLA is a vision-language-action model designed to use learned future-scene information when generating robot commands without running the video model during control.
- Learning future-scene clues
PDRVLA’s Physical Dynamics Module extracts 50 physical tokens from intermediate vision-language features. It learns these tokens from future image, depth, and motion targets, with additional guidance from a frozen video model during pretraining.
The action expert combines the tokens with final-layer features and the robot’s current state. In joint policy training, direct future prediction continues, but the video model is removed.
- A prediction comparison
A direct comparison tested whether adding video-model guidance improved prediction beyond direct future-feature training. On the same 828 RoboCOIN clips from 100 episodes excluded from training, the equally weighted mean normalized prediction error across image, depth, and motion was 0.6354 with the added guidance and 0.6439 without it.
This result supports a prediction improvement in that pretraining comparison, not an improvement in action success.
- Performance in simulation
The paper also reports task success on four simulation tests: LIBERO for single-arm tasks, LIBERO-Plus for those tasks under observation and instruction changes, RoboTwin 2.0 for two-handed tasks, and SimplerEnv for WidowX visual-matching tasks.
PDRVLA had strong reported averages against published methods, but did not lead on every task. For example, it scored below a comparator on one SimplerEnv task.
- What the evidence supports
The prediction comparison and task tests answer different questions: one measures future-feature error, while the others measure whether robot tasks succeed. The policy results use benchmark-specific adaptation, and baseline scores come from their own published protocols.
Because the reported policy evaluations are simulations, they do not establish real-robot performance.
Deep
- Original paper authors
- Hanqing Wang, Zhaotian Cai, Ruichuan An, Bohan Zeng, Tao Zhang
- Submitted to arXiv
- License
- arXiv non-exclusive