A robot controller repairs harmful visual futures and skips needless ones
Utility-WAM: Correcting Future-Induced Action Changes · arXiv:2610.39367
In plain words
Robot-control systems can use a model’s imagined future video to choose movements. That can help on some tasks, do little on others, or make precise movements worse, while requiring extra computation.
Utility-WAM corrects action changes linked to the imagined video, then chooses whether to use that video-based route or a route that skips video. In RoboTwin 2.0 simulation, selective use raised average task success to 93.3%, compared with 92.8% for always-on correction.
The research
When a robot uses generated video to anticipate what happens next, does that extra visual information reliably improve its actions? It can help on some tasks, have little effect on many, or sharply hurt precision tasks while adding computation. It remains unclear how to retain useful foresight without accepting its harmful action changes.
Utility-WAM corrects action changes caused by generated robot video, then learns when to use the corrected video route or skip it.
The authors compared the same robot model using either a future-free action route or one conditioned on generated video, with matched starting scenes and random seeds.
On LIBERO, a simulated object-manipulation test, video raised average closed-loop task success by 0.8 points; on RoboTwin 2.0, a simulated bimanual manipulation test, it lowered the average by 0.8 points.
Those averages hid sharp losses on two RoboTwin 2.0 precision tasks. Correction reduced the losses, and selective routing raised average success to 93.3%, versus 92.8% with always-on correction. This supports correcting and selectively using generated futures in the tested tasks.
These results come from the tested simulation tasks and three physical tasks. The evaluations use a common pretrained model checkpoint, so variation across independently retrained backbones is unknown; the gate is trained from action-error comparisons rather than task success.
What the evidence shows
- The uneven value of futures
Utility-WAM is a robot-control method that addresses a specific tension: generated future video can guide actions, but it can also distort them and costs extra computation. It first corrects action changes linked to video, then learns whether to use that corrected route or a future-free route. The authors evaluate the approach in simulation and on physical robot tasks.
- Small averages hide sharp losses
The authors compared the same shared model with and without action access to generated video. LIBERO tests simulated object-manipulation tasks, and its reported score is average closed-loop task success; video raised that average by 0.8 percentage points.
RoboTwin 2.0 tests simulated bimanual manipulation tasks, and video lowered average success by 0.8 points. These averages use matched starting states and random seeds, but conceal how task effects vary.
- Correction targets the damage
The RoboTwin 2.0 task-level comparison identified severe losses on Press Stapler and Place Can Basket: success fell by 54 and 34 percentage points, respectively, compared with the future-free route. Each task’s change averages clean and randomized conditions.
With always-on correction, the losses shrank to 6 and 14 points. On LIBERO-Long, average success was 98.4% with correction versus 97.8% for the uncorrected video-conditioned route.
- A control tests the objective
A control compared the full adapter objective with an action-only adapter on RoboTwin 2.0. Both used future video, kept the shared model frozen during adapter training, and disabled the gate.
The action-only adapter reached 91.4% average success from 91.1% without an adapter; the full objective reached 92.8%. This supports the value of the full objective in this setup, beyond adding an adapter alone.
- Selective use and physical trials
The gate chooses whether to generate video before running either action route. On RoboTwin 2.0, it raised average success to 93.3% from 92.8% with always-on correction, generated video for 24.5% of action queries, and lowered measured latency by 37.9% versus always-video inference.
On three physical bimanual tasks, Utility-WAM succeeded in 53 of 60 trials, compared with 43 of 60 for FAST-WAM.
Deep
- Original paper authors
- Feihong Zhang, Xi Lin, Yulong Shi, Zuxing Lu, Yanghong Mei
- Submitted to arXiv
- License
- arXiv non-exclusive