RynnUnify links video prediction with long-horizon planning
RynnUnify: Towards Unified Embodied Brain · arXiv:2610.32965
In plain words
The paper introduces RynnUnify, a model that reasons about the physical world and predicts video to help plan a task across multiple steps. It plans a subtask, predicts what the scene may look like afterward, then uses that prediction to plan what comes next. An additional model can use predicted video to answer questions about unseen future states.
On a test of whether long visual plans fit the task and earlier steps, RynnUnify scored 9.75 points above Cosmos 3 Nano. The paper also reports robot-control results on selected simulated tasks and three real-robot tasks.
The research
When a robot's next decision depends on changes that have not happened yet, can a model connect its plan to a useful prediction of what the scene will look like? That is difficult over many steps because prediction errors can accumulate and static visual predictions can miss motion between them.
RynnUnify links a reasoner's subtask plans to video predictions, using each predicted outcome to inform the next step in a long visual plan.
The authors introduce Rynn-VP, a test of whether visual plans stay consistent with the task and preceding operations over long sequences. RynnUnify scored 9.75 points above Cosmos 3 Nano on its overall measure.
This supports stronger performance on that planning test, though the supplied account does not specify the scoring details.
Rynn-GEU asks about action consequences, physical changes, and spatial relationships not visible in the input. On the same questions, the agentic extension's average score was 6.0 points above direct reasoning, supporting video-assisted reasoning for these unseen states.
The score comparisons are limited to the evaluated models and questions; the supplied passages do not give full scoring or training details. Robot-control results cover 20 selected simulated tasks and three real-robot tasks, not other tasks or robots.
What the evidence shows
- Planning across changing scenes
Long robot tasks require more than recognizing a scene: a model must anticipate physical changes and use them in later decisions. The paper identifies a challenge in keeping those steps connected, since prediction errors can accumulate and static visual predictions can miss intermediate motion.
- Linking plans to predictions
RynnUnify is a unified embodied model that pairs a reasoner with a video generator. The reasoner plans a subtask and describes its expected visual evolution; the generator predicts a video, and that outcome provides context for the next subtask. The model therefore links plans to predicted scene changes across successive steps.
- Testing long visual plans
To assess long-horizon visual planning, the authors introduce Rynn-VP, a test of whether plans remain consistent with the task and earlier operations.
RynnUnify's overall score was 9.75 points higher than Cosmos 3 Nano's. This is evidence of stronger performance on that test, while the supplied passages leave other scoring details unspecified.
- Reasoning about unseen states
For questions about states not directly visible in an input, Rynn-GEU tests reasoning about action consequences, physical changes, and spatial relationships. The authors' agentic extension can answer directly or use video generated by RynnUnify.
On the same questions, its average score was 6.0 points above direct reasoning, supporting a benefit from video-assisted answers in this test.
- Testing robot control
A world-action extension adapts the full model for robot control. It succeeded on 51.5% of 20 selected composite RoboCasa365 tasks in simulation, and averaged 46.7% success across three real-robot tasks.
These results show performance in the reported settings, with the simulated result covering selected tasks rather than the entire collection.
Deep
- Original paper authors
- Minghao Zhu, Bohan Hou, Kun Xiang, Zixuan Wang, Yinzhou Tang
- Submitted to arXiv
- License
- CC BY 4.0