A robot’s misread instruction can be written into its weights
Distilling Instructions into Weights: Closed-Form Edits Fix a Robot Policy's Misreading · arXiv:2611.11894
In plain words
A robot can sometimes do a task when the instruction is phrased one way, but drop an object when it is phrased another way. This paper introduces a method that finds wording the robot follows, then copies that wording’s effect into the robot’s weights without changing the instruction.
On one simulated cube task, the edited robot succeeded in 106 of 128 held-out episodes, compared with 27 for the unedited robot. The result applies to that task and checkpoint.
The research
A robot that drops an object despite having the needed skill presents a different repair problem from one that cannot perform the motion. Here, success under rewording makes an instruction-reading error a possible target for a weight edit, while the original instruction stays in place.
Closed-form instruction distillation edits a robot policy’s weights to carry a working instruction’s effect into the original wording.
In replay, the robot released the cube in all 21 recorded cases with the original wording and after the cube was renamed; moving the destination phrase next to “cube” reduced releases to one to three. The authors read this as a wording misbinding, then used the working wording to edit the weights.
On identical held-out simulator starts, the original instruction stayed fixed while success rose from 27 of 128 episodes to 106. The comparison supports a task-specific improvement from the edit.
The evaluations used one zero-shot checkpoint in simulation, with 16 to 128 paired episodes per comparison, and failure levels were assigned without an independent rater. The released tool was evaluated on one task.
What the evidence shows
- A wording error can look like missing skill
Closed-form instruction distillation is a method that edits a robot policy’s weights without gradients or training data. It aims at a specific failure: a zero-shot robot policy drops a cube before reaching the bowl under one instruction, but succeeds with a rewording.
That contrast makes a wording error a plausible diagnosis, rather than evidence that the policy lacks the task skill.
- Rewording tests the diagnosis
In an offline replay of 21 recorded premature-release decision points, the policy dropped the cube in all 21 replays with the original wording and after the cube was renamed. When “in the bowl” was moved next to “cube,” only one to three replays still released it.
The images, robot state, and action-generation noise stayed fixed while the instruction changed, so the result supports the researchers’ interpretation that the original wording linked the destination to the wrong object.
- The edit improves held-out success
The method writes the effect of a known-good wording into the policy’s weights while keeping the task instruction fixed. The paired evaluation supports a task-specific improvement in success from that edit on held-out simulator starts.
- Update direction matters in the control
The randomized update performed closer to the unedited policy than to the distilled edit, even though its overall scale and other listed properties matched.
This supports the interpretation that the distilled update’s directions mattered in this control, while the difference in changed values limits what the comparison can isolate.
- Transfer is limited and placement matters
The unchanged edit improved success on two other tasks: eyeglasses pickup rose from 22 to 44 successes out of 64 held-out starts, and putting the cube left of the bowl rose from 8 to 36.
In the placement comparison, the vision-language model location performed best on the original task and was the only tested location that transferred to the two other tasks in that comparison.
- Fixing one decision may not finish the task
The stacking edit reduced the targeted wrong action, but task success changed little in development and showed no statistically significant difference on held-out starts. The remaining failure involved placing the third cube, which the edit did not address.
Deep
- Original paper authors
- Ali-Adeeb Abbas, Esmaeil Seraj
- Submitted to arXiv
- License
- CC BY 4.0