RoboAuto grounds robot programs in physical scene knowledge
RoboAuto: Autonomous Robot Program Search through Multi-Level Physical Understanding · arXiv:2610.36533
In plain words
Robot programs can keep getting revised without solving the physical problem: a robot might use the wrong grasp or move on before checking whether an earlier action worked.
RoboAuto is a framework that helps a coding agent search for robot programs using reusable object knowledge, measurements of the current scene, and checks of each stage’s result.
In selected multi-stage simulation tasks, RoboAuto’s programs succeeded more often on average than programs found by ASPIRE. The comparison covers ten task and perturbation settings, so it does not establish results for every robot task.
The research
When robot programs are revised after failures, can search learn what interaction suits an object, measure whether that strategy fits the current scene, and verify that each action worked?
RoboAuto tests a system that combines those steps, asking whether the resulting programs succeed on selected simulation tasks and transfer tests.
RoboAuto combines reusable object knowledge, measurements of the current scene, and staged outcome checks to search for robot programs.
RoboAuto searches for robot programs by combining reusable physical knowledge with measurements of the current scene and checks that actions achieve their intended effects.
The question is whether those elements help the search find programs for tasks that depend on object interactions and scene geometry.
Across selected multi-stage simulation settings, RoboAuto had higher mean task success than ASPIRE. An ablation also showed rising success as components were added, with the largest increase when scene grounding joined the knowledge base.
This supports the system’s combined approach, while the controls show that the knowledge base alone does not explain its full result on the tested tasks.
These results cover selected simulation settings and three real-robot tasks. The real-robot measure gives credit for partial stage completion, and a human operator resets the scene and keeps the robot and surroundings safe; the study therefore does not demonstrate autonomous real-world operation.
What the evidence shows
- When revisions miss the physics
RoboAuto is a framework for autonomous robot program search. It addresses a problem illustrated in baseline runs: repeated program revisions did not resolve some physical uncertainties, such as whether a grasp strategy suited an object or whether a prior action had activated a stove.
These examples motivate a search process that reasons about interactions and checks what actually changed in the scene. [F2].
- Knowledge tied to each scene
RoboAuto separates reusable object knowledge from executable skills, which are verified robot programs. It grounds the retrieved knowledge in the current scene by identifying task-relevant objects and measuring action-critical quantities.
For long tasks, it searches through stages and checks each intended physical outcome before proceeding; when execution leaves a cause uncertain, it can run a separate targeted experiment. [F4].
- Higher success on selected tests
On LIBERO-PRO Long, a test of tasks with dependent physical changes that must happen in order, RoboAuto’s mean success was 73.4% across ten selected settings, compared with 30.8% for ASPIRE. On LIBERO-Plus, which tests camera-view and layout changes, the means were 90.3% and 44.7% across six selected settings.
These are comparisons between complete methods, so they show a performance difference without identifying one component as its cause. [F7].
- What the components add
The component ablation provides a separate view: on five tasks, mean success rose as search stages and experiments, reusable knowledge, and then scene grounding were added. A control adding the knowledge base to ASPIRE raised its mean only from 36.8% to 43.4% on ten short-horizon tasks, while full RoboAuto reached 84.8% there.
Together, these results suggest that the knowledge base alone does not account for the full system’s performance in that control. [F9].
- Transfer has defined bounds
A transfer control tested LIBERO-derived knowledge on four RoboTwin 2.0 tasks using a different simulator and bimanual robot. Mean success was 91.5% with that knowledge and 24.5% without it, under the same search protocol.
The evaluation also included three real-robot tasks, but those used a partial stage-completion measure and a human operator for scene resets and safety. [F10][F11][F12].
Deep
- Original paper authors
- Buzhao Liu, Jianing Ye, Xiu Yuan, Yangtong Feng, Chengsong Huang
- Submitted to arXiv
- License
- arXiv non-exclusive