In plain words
A robot controller needs a useful preview of what will happen when it moves, especially when its gripper touches or moves an object. PhysCanvas is a model that imagines camera and touch observations after each robot action, then uses those imagined interactions to assess or train controllers.
Across four robot tasks, adding task-outcome guidance made imagined success rates agree more closely with real results. Training a tested controller through imagined interactions also raised its average real-world success, though results varied by task.
The research
When a robot practices or is assessed in an imagined world, how can we tell whether its predicted contact and task result are useful guides to what will happen on the real robot? PhysCanvas addresses this by linking predictions of camera views, touch, and task outcomes to the robot's actions.
PhysCanvas predicts camera, touch, and task outcomes from robot actions, then uses imagined interactions to assess and train robot controllers.
PhysCanvas predicts camera, touch, and task observations after robot actions. To see whether its imagined outcomes are useful, the authors compared success rates from controller rollouts in the model with real robot results across four tasks.
Adding task-state supervision raised the mean task-wise correlation from 0.602 to 0.885. The authors report that image reconstruction measures changed in mixed directions, supporting their interpretation that the gain reflects better task-outcome modeling.
Separately, imagined GRPO training raised pi0.5's equal-weight real-world success mean from 59.1% to 70.8% with M4.
These results cover four real-world manipulation tasks and the tested policy families. Average gains do not mean every task improved: for example, pi0.5's bottle success rate was lower with M4 than with M1, and FM remained at zero on two tasks.
Analysis
Deep
- Original paper authors
- Zebin Xing, Yuhang Zheng, Yupeng Zheng, Yichen Zhang, Zichen Li
- Submitted to arXiv
- License
- arXiv non-exclusive