A robot data system makes simulator motion look like real scenes
PyRUA-DataFarm: LLM Agent as a Cost-Efficient Teleoperator for Real-Ready Robot Data · arXiv:2611.05320
In plain words
Robot learners need examples that show both what a robot did and what its cameras saw. DataFarm takes a task request and a short recording of the real scene, then has software build a simulated task and generate robot motions with exact action labels. A video model makes those motions look like the real scene, and checks screen the results.
On five tasks with one robot, policies trained on DataFarm examples did about as well as those trained on teleoperated examples in the original scene, and better when the tablecloth and lighting changed.
The research
Robot training data must show what the machine did as well as what it saw. A system that can create checked examples without collecting new teleoperated demonstrations could change whether teams need to choose between exact simulated actions and images that resemble their real workspace.
In these five tasks, the results also suggest that varied visual training can matter when appearances change.
DataFarm turns a task request and one scene recording into simulator-labelled robot episodes rendered with real-scene appearance, then checks them before training.
Researchers trained policies further on equal-sized datasets of 100 episodes per task, then tested each in 20 trials per task and condition. Across five tasks, farm-trained policies recorded 93 successes out of 100 in the original scene, versus 84 for teleoperation-trained policies. Under an unseen tablecloth and changing room light, the totals were 87 versus 46.
A control using the same grounding, expert, and simulated actions but direct simulator images recorded 26 and 21 successes in those conditions. This comparison supports the role of how observations are produced in the tested setup.
The real-robot comparison covered five tasks on one dual-arm robot, and the starting policy checkpoint is unspecified. The results also depend on whether the simulator can reproduce the skills and dynamics a task requires.
What the evidence shows
- The data problem
DataFarm is a robot-data generation system that turns a task request and one recording of the target scene into simulated episodes with action labels and realistic camera views. It addresses two linked obstacles: building a new simulated task can take human engineering, and simulator images may look unlike the real scene.
Video generation can improve appearance, but generating motion itself may break the exact link between actions and labels.
- Separate motion and appearance
The system separates motion from appearance. A coding agent builds the task and an expert controller; the simulator generates trajectories and records their actions and states.
A video model renders those trajectories using real camera images, while automatic checks and an independent reviewer screen each episode. This lets the rendered images change while the simulator remains the source of the robot’s labelled motion.
- The real-robot result
Across the five tasks, DataFarm and teleoperation performed similarly in the original scene, while DataFarm had more successful trials under the tested tablecloth and lighting changes. These totals summarize 20 trials per task on one dual-arm robot; they do not mean every task improved.
- Why rendering matters
Because the control shared DataFarm’s task grounding, expert, and simulated actions, its lower success totals point to how training observations were produced as an important difference in these tested conditions. The comparison does not show that realistic rendering alone explains the gap in every setting.
Deep
- Original paper authors
- Kewei Zhang, Juncheng Ma, Guodong Lv, Duomin Wang, Feixiang Ruan
- Submitted to arXiv
- License
- CC BY 4.0