EN
A motion-alignment score tracks robot models’ task success
Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success · arXiv:2610.31670
In plain words
Testing a robot model by letting it try tasks can be slow. Sensorimotor Alignment (S²) is an offline score that compares what a model sees and reads with movements recorded in robot demonstrations.
Across 12 trained robot models, higher scores went along with higher success rates on a simulated manipulation test. The score may help compare models before running them, but these results do not show that raising the score makes a robot better.
The research
Can a robot model’s likely task performance be estimated before the slow process of letting it attempt tasks? Existing image-understanding and video-quality scores do not directly measure whether a model’s representations reflect the movements a robot needs.
Sensorimotor Alignment compares robot models’ sensory-feature neighborhoods with recorded movement neighborhoods to assess their alignment offline.
The authors compared S² with downstream task success across several robot model families. In SimplerEnv, which tests simulated robot manipulation and reports the rate of successful tasks, S² was positively associated with success across 12 fine-tuned models.
It also tracked the later success of robot models trained from eight raw vision-language backbones, where aggregate visual question answering scores showed virtually no association.
Associations also appeared for world action models and physical-robot policies, though the physical-robot result depended on using an exploratory movement horizon. Together, these observations support S² as an offline measure that can help identify promising models for further testing.
These results show associations across the tested models, not that increasing S² will improve a robot’s success.
In the physical-robot evaluation, the stronger association appeared only with an exploratory trajectory horizon; the prespecified horizon did not show a statistically significant positive association.
What the evidence shows
- A gap in robot evaluation
Robot models are often judged by letting them perform tasks in simulation or in the real world, but those tests can be slow and noisy. The paper introduces Sensorimotor Alignment (S²), a training-free offline metric, to ask whether a model’s image-and-instruction representations reflect the movements needed for robot tasks.
Unlike visual question answering or video-quality scores, it compares sensory representations with recorded motion.
- Matching representations to movement
S² compares the nearest neighbors of each episode in two spaces: one built from a model’s image-and-instruction features, the other from recorded robot movements. The authors tested whether this overlap tracks task success.
In SimplerEnv, a simulated robot manipulation test that measures the rate of successful tasks, higher S² went with higher success across 12 fine-tuned vision-language-action models.
- Testing another offline score
The association also appeared for eight raw vision-language model backbones: their S² scores, measured before robot training, tracked the later SimplerEnv success rates of the robot models trained from them. Aggregate visual question answering scores showed virtually no association with those success rates.
This comparison suggests that a score connected to motion can be more informative for these models than a general image-and-language score.
- Where the evidence points
Evidence across other settings was encouraging but qualified. S² tracked simulated task success across six world action models, and it tracked five policies’ physical-robot success at an exploratory movement horizon.
At the prespecified horizon, the physical-robot association was weaker and not statistically significant. The authors also found that image features tracked success more closely than instruction features in fine-tuned models, while a small freezing control linked lower alignment with lower success.
Deep
- Original paper authors
- Yunfei Xie, Howard H. Qian, Shiyi Lan, Kaiyu Hang, Chen Wei
- Submitted to arXiv
- License
- arXiv non-exclusive