A generalist model adapts while controlling a robot
On Agentic Manipulation Policies · arXiv:2611.09892
In plain words
Some robot systems ask an AI model to choose from ready-made skills, while others have it write instructions before the robot starts. This paper introduces AMP1, a setup where a generalist model writes code to control a robot directly, then uses new camera views to change what it does next.
In simulated pick-and-place tasks, AMP1 with Gemini 4 Argon succeeded more often than the listed specialist robot systems. Physical trials also succeeded on many tasks, though the model can be slow and cannot change a motion while it is happening.
The research
A robot that can only call fixed skills has limited ways to respond, while one following a prewritten program cannot rethink its strategy mid-task. Direct control with observation-based adjustments offers another approach.
The results make success on simulated tasks and physical trials a relevant test of that approach, subject to the study’s specific setups.
AMP1 keeps a generalist model in direct robot control, letting it write motion code, observe the result and revise its next actions as a task unfolds.
On RoboLab-120, which tests simulated, language-described pick-and-place tasks, AMP1 with Gemini 4 Argon succeeded in 85.6% of episodes; the strongest listed vision-language-action policy succeeded in 49.2%. The leaderboard policies’ training procedures and starting states were not specified as matched.
Physical trials and simulated tasks with persistent memory provide additional evidence: AMP1 succeeded in 104 of 108 recovery-enabled robot trials, and practice improved success on unseen layouts. Together, the results support direct, adaptable control in the tested settings.
The robot often waits while the model reasons, and it cannot revise a motion while that motion is underway. The evaluations largely use nearly still scenes, so they do not show how well AMP1 responds to moving targets or slipping objects.
What the evidence shows
- Keep the model in control
Agentic manipulation policies (AMPs) keep a generalist AI model in the control loop: it directly commands a robot manipulator and can revise its actions as new observations arrive. This addresses a limitation in two other approaches.
A model choosing from pre-existing grasping or placing skills is restricted by those skills, while a model that writes a control program before execution cannot rethink its overall strategy as the task unfolds.
- A higher simulation result
On RoboLab-120, a test of simulated, language-described pick-and-place tasks, AMP1’s episode success rate was higher than the leading listed vision-language-action policy’s. Because the leaderboard methods’ training procedures and starting states were not specified as matched, this comparison shows reported performance on that test, not an isolated effect of AMP1’s control approach.
- Trials beyond simulation
The physical trials show AMP1 could complete varied tasks across three robot instances under a protocol that let it recover from controller errors and continue. That recovery rule matters: the result reflects performance with recovery enabled, not the same conditions as the simulation comparison.
- Memory carries experience
The practice result suggests saved experience can transfer to unseen layouts while model weights stay fixed. The separate across-task result is more tentative because it comes from a single pass through one random task order.
- Less input costs more work
With Opus 5.5 and all cameras, the RGB-only condition retained much of the performance seen with full visual input, while requiring more model calls per episode.
This suggests the model could reconstruct some missing geometry in this RoboLab-120 setup, at a cost in success and efficiency; weaker models were more sensitive to sensory loss.
Deep
- Original paper authors
- Chen Chen, Omid Bagherieh, Naveen Nikamanth Appiah Balaji, Jesse T. Gonzalez, Negin Heravi
- Submitted to arXiv
- License
- Not specified by source