DexForesight transfers hand-and-scene foresight into robot actions
DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight · arXiv:2611.11166
In plain words
DexForesight is a robot-control method that teaches a policy to use predictions about how a hand and its surroundings will move. From demonstrations, a world model learns hand motion and changes in scene visuals; the policy then learns to use those features while learning actions.
When acting, the policy uses current camera images, language, and robot state without running that model.
In tests, it had higher overall task success than the strongest reported comparison in both 11 simulated tasks and three physical-robot tasks. Those results cover these settings, not every dexterous task.
The research
A dexterous robot must choose movements while parts of its hand may be hidden, and generating full future images adds work during control.
This result changes the design question: predictive information about hand and scene motion can be learned during training and carried into the acting policy, rather than recomputed as future observations.
DexForesight transfers action-conditioned hand-and-scene predictions into a robot policy, so it can use learned foresight without generating futures while acting.
The method first trains a spatial world model to predict hand-point motion and changes in scene visuals under demonstrated controls. It then freezes that model and aligns its predictive features with policy queries while the policy learns actions.
At deployment, the policy uses current images, language, and robot state without running the world model.
Across 11 simulated tasks, overall task success was 60.5%, versus 50.9% for Fast-WAM; across three physical-robot tasks, it was 58.3%, versus 50.0% for WLA. These separate comparisons show higher reported success than the strongest listed baseline in each setting.
The external methods also differ in architecture and training, so these comparisons do not isolate the transfer step. The physical-robot result covers three selected tasks, with 20 trials each.
What the evidence shows
- A policy learns spatial foresight
DexForesight is a robot-control method that transfers predictive features from an action-conditioned spatial world model into a policy that generates actions. The method addresses a challenge in dexterous manipulation: a controller must account for how fingers and nearby objects move, including fingers hidden from cameras.
Its design learns those changes during training, so the policy can use the resulting features when acting.
- Actions guide the predictions
The world model predicts point-by-point hand motion and changes in scene visuals under demonstrated controls. This gives the training process information about how particular actions change the hand–scene arrangement, including motion of occluded fingers.
The distinction matters because recognizing only the current visible scene does not tell the controller how a movement will reshape that arrangement.
- Prediction moves into the policy
DexForesight freezes the trained world model and aligns its predictive features with queries inside the action-generating policy. During deployment, that policy uses current images, language, and robot state; it does not run the world model, reconstruct depth, or generate future observations.
The learned prediction therefore informs action generation without requiring future decoding while the robot acts.
- Higher success in simulation
Across 11 DexJoCo simulation tasks spanning single-arm and bimanual manipulation, DexForesight’s overall task success rate was 60.5%, compared with 50.9% for Fast-WAM, the strongest reported baseline overall.
This is a higher reported success rate in that simulation suite; the external methods’ training conditions were not established as matched.
- A result on physical robots
Across three physical-robot tasks, DexForesight’s overall task success was 58.3%, compared with 50.0% for WLA, the strongest reported baseline overall. This is a higher reported rate on these selected tasks, though the methods differed in architecture and training.
- Features retain motion information
A probe tested whether future hand-motion information could be read from the policy’s features alone, after removing the world model and its prediction experts. At a 0.54-second horizon, its error was 10.01 mm for DexForesight and 13.59 mm for an architecture-matched action-only policy. This measures how well the features reveal future hand movement, not task success.
Deep
- Original paper authors
- Zhongxi Chen, Yi Yang, Xinyi Chen, Mingzhe Ye, Yaping Li
- Submitted to arXiv
- License
- arXiv non-exclusive