Robot policies improve when they can read earlier visual features
From Pixels to Actions - Towards Native Vision-Language-Action Policies · arXiv:2611.06918
In plain words
Robot arms need to connect instructions with details in an image, such as where an object is and how it is positioned. Many systems pass images through a separate vision module before deciding what to do.
NEO-VLA processes image pieces and instruction words together, then lets its action-making component use information from different depths of that shared system.
In tests across four robot task suites, it averaged higher success than two comparison systems when their shared backbones were frozen, though the backbones also differed in pretraining and size.
The research
A robot’s action can depend on visual details that disappear when an image is reduced to one fixed representation.
The results make access to earlier visual processing a concrete design question: in these robot task tests, policies that could use it performed better than a late-only readout, although the comparisons do not isolate encoder removal.
NEO-VLA is a robot manipulation policy that processes images with instructions and lets its action expert draw on visual information from multiple backbone depths.
On LIBERO’s four robot task suites, researchers compared NEO-VLA with Qwen3-VL-2B and Gemma-3-4B after training only each action expert. NEO-VLA averaged 90.9% success, versus 79.5% and 84.5%.
The shared pretraining setting and matched action training make the comparison informative, but backbone pretraining and capacity also differ. A separate layer test found higher averages when the expert could read across the backbone than when it read only later layers.
This supports the value of access to early visual computation, while a control in Qwen3-VL shows that direct access to intermediate features alone did not reproduce NEO-VLA’s frozen-backbone result.
The frozen-backbone comparison also changes the pretrained backbone and its capacity, so it cannot show that removing a separate vision encoder alone caused the difference. The results come from the tested task suites and model settings.
What the evidence shows
- Keep visual processing accessible
NEO-VLA is a vision-language-action policy for robot manipulation. It addresses a design question: should a robot action system see only a fixed summary from a separate vision encoder, or can it use visual information as it is processed?
In LIBERO, a set of robot manipulation tasks scored by the share of successful episodes, NEO-VLA lets its action expert read information from selected depths of a shared image-and-language backbone.
- Compare frozen backbones
The frozen-backbone comparison suggests NEO-VLA’s shared visual pathway gives its action expert useful representations before backbone adaptation: its average was 90.9%, versus 79.5% for Qwen3-VL.
Because backbone pretraining and capacity also differ, this comparison cannot attribute the gap to removing a separate vision encoder.
- Earlier layers help control
When both systems retain the same 40-layer backbone and 28-layer action expert, reading layers across the backbone outperformed reading only layers 12–39: 90.9% versus 86.7% average success with the backbone frozen. This comparison supports making earlier layers available to the action expert.
- Probe information and its route
A probe on diagnostic models swapped the information entering one expert attachment point at a time. Swaps at layers 0–10 increased translation prediction error most, while noise matched to the swap’s magnitude kept error within 0.004 cm of the unchanged case.
A separate Qwen3-VL control found that direct access to intermediate vision features did not reproduce NEO-VLA’s frozen-backbone result. Together, these tests support the role of visual information and its presentation, not a claim that encoder removal alone explains the main comparison.
- Check performance under shifts
On LIBERO-Plus, which tests robot task success under seven kinds of change, NEO-VLA had the highest task-weighted total among listed models with both code and weights public. Its scores were stronger for camera and language changes than for robot and layout changes.
This extends the findings to tested shifts in visual observation, but the comparison is cross-model and the source does not establish matched training or evaluation conditions for every listed model.
Deep
- Original paper authors
- Weichen Fan, Haiwen Diao, Penghao Wu, Siyuan Hu, Ziqi Huang
- Submitted to arXiv
- License
- CC BY 4.0