EN
Robot learning uses video to ground motion reasoning
ThinkWorld: World Imagination Reinforces Chain-of-Dynamics Reasoning · arXiv:2610.37666
In plain words
ThinkWorld is a robot-learning method that uses a shared set of motion tokens to represent movement in human videos and robot demonstrations. It predicts what the scene will look like after a movement and learns from how that prediction matches the video.
On LIBERO-PRO, a robot-manipulation test that changes objects, positions, instructions, goals and environments, it reported 55.6% average success after training on standard LIBERO without those shifts. This is evidence about that test setup, not every robot task.
The research
Robot demonstrations tell a robot what actions to take, but cover fewer behaviors than human videos, which do not show executable robot actions. It remains unclear whether those videos can improve robot motion reasoning while keeping it tied to changes that actions cause in the world.
ThinkWorld gives human and robot motion a shared token representation, then trains motion reasoning to favor imagined futures that match video.
LIBERO-PRO tests robot manipulation under controlled changes to objects, positions, instructions, goals and environments; its score is average success across benchmark axes. ThinkWorld-LIBERO was trained further on standard LIBERO, then evaluated without exposure to those shifts, and reported 55.6% average success. That result shows performance under the tested shifts in this training setup.
A separate pretraining study found higher LIBERO-PRO success as the human-video share rose while robot-data quantity stayed fixed; a matched-scale robot-only control helps distinguish that result from gains due to scale alone.
The reported training results use one training seed, so episode-level confidence intervals do not show how results vary across independently trained models. The human-video comparison studies also use configurations that differ from the final model.
What the evidence shows
- Bridge two kinds of experience
ThinkWorld is a vision-language-action learning framework for robot control. The problem it addresses is that robot demonstrations show executable actions but cover a limited range of behavior, while human videos offer broader experience without robot actions.
Motion inferred from video can also reflect camera or appearance changes instead of controllable movement. The framework aims to connect the broader video experience to action reasoning grounded in motion.
- Give motion a shared form
The framework represents intended human and robot motion with a shared sequence of discrete tokens called Chain-of-Dynamics, or CoD. It learns this representation from robot demonstrations and annotated human hand motion.
CoD tokens are not executable robot commands: a separate action expert predicts the robot actions. This shared representation gives the model a common way to reason about motion across the two sources.
- Learn from observed outcomes
ThinkWorld also learns from what videos show happens after movement. It samples alternative CoD sequences, predicts the future visual features for each, and rewards candidates whose predictions better match the observed future.
Because a video provides this transition even without action labels, the training signal can draw on human and robot videos without executing candidate motions.
- Measure performance under shifts
LIBERO-PRO tests robot manipulation under changes to objects, positions, instructions, goals and environments, and reports average success across its test axes. ThinkWorld-LIBERO was trained further on standard LIBERO, then evaluated without exposure to those shifts; its reported average success was 55.6%. This supports transfer to the shifts in that test setup, rather than a claim about all new tasks.
- Check the role of human video
In a separate pretraining study, robot-data quantity was fixed as the human-video share increased, and all variants were trained further on standard LIBERO before evaluation. The 50% human-video variant scored about 1.2 percentage points above a robot-only control matched for training scale and iterations.
The study also reports qualitative robot rollouts composing full task sequences shown in human video, although robot demonstrations covered only individual skills.
Deep
- Original paper authors
- Chaehyun Kim, Chi-Pin Huang, Jisoo Kim, Min-Hung Chen, Yu-Chiang Frank Wang
- Submitted to arXiv
- License
- CC BY 4.0