Robots learn tasks through simulator feedback in context
World In-Context Learning for Agentic Robot Manipulation · arXiv:2611.12885
In plain words
A robot can copy a task from a demonstration, but a demonstration alone does not show what will happen in a new scene. This paper introduces a way for a model to practice in a simulator and save what it learns as text, while keeping its model weights unchanged.
In a 50-task simulator evaluation, RoboScript learned from one demonstration per task and practiced only in clean scenes. Its success stayed similar in clean and randomized scenes, though the result applies to this test setup.
The research
A robot that learns only from a demonstration may not know how a different scene will affect its actions.
Simulator feedback offers a way to record what happens during practice and reuse those lessons in new scenes, changing what a one-demonstration approach can be expected to handle in this tested setting.
RoboScript implements World In-Context Learning: a frozen model practices with simulator feedback, stores lessons in text, and uses them on new robot scenes.
In RoboTwin 2.0, RoboScript rehearsed each task in clean simulator scenes, then faced new scenes without further feedback. Across the 50 tasks, its GPT-6.1-Sol version scored 83.4% mean success, versus 61.7% for 4D-WAM, a trained action policy. The rates were close in clean scenes, but diverged in randomized ones.
That pattern supports the claim that simulator practice can produce task knowledge useful in changed scenes. The comparison does not isolate learning procedure or number of demonstrations.
The main comparison is specific to RoboTwin 2.0 and uses different learning procedures, demonstration counts, and evaluation episode counts for RoboScript and 4D-WAM.
The physical-robot test covered four tasks, used planned action sequences without online replanning, and does not show how the method performs on other tasks or hardware.
What the evidence shows
- Learning needs world feedback
World In-Context Learning (WICL) is a learning paradigm that adds feedback from a world to in-context learning. RoboScript implements it for robot manipulation: a model starts from a demonstration, practices in a simulator, and records lessons in text while its weights stay frozen. The question is whether those lessons can carry into scenes the model has not practiced in.
- The randomized scenes decide
On RoboTwin 2.0, RoboScript rehearsed on clean scenes and had similar success in clean and randomized scenes. Its advantage over a trained action policy came mainly in randomized scenes, though the two systems used different learning procedures and numbers of demonstrations.
- A test of simulator feedback
On a separate 10-task test, simulator interaction during memory learning improved success compared with writing the memory from the same demonstration alone. This supports a contribution from world feedback in that test, while the result does not establish its size across all 50 tasks.
- A small hardware transfer
The paper also reports a physical-robot transfer test: after an adapter remapped robot-specific parameters, memories rehearsed in simulation were used on a dual-arm PiPER robot. They succeeded in 10/10, 7/10, 7/10, and 6/10 episodes across four tasks. This shows a limited transfer example; the planned action sequences did not replan online.
Deep
- Original paper authors
- Heng Fang, Zehui Guo, Yuehan Zhang, Mingyang Du, Dingkang Liang
- Submitted to arXiv
- License
- arXiv non-exclusive