Robot training aligns 2D vision with 3D spatial knowledge
SLA: Scaling Spatial-Language-Action Model via Continual Pre-training · arXiv:2611.08373
In plain words
Robot policies need to understand where things are to move them accurately, but many learn mainly from flat images, and robot datasets do not all include depth or camera measurements. SLA is a way to train a robot model by matching its visual features to spatial information from a frozen 3D model while it learns actions.
In one simulated test with varied scenes, SLA had a higher average task success rate than the π0.5 comparison model. That result covers the tested tasks and setting.
The research
Precise manipulation depends on spatial understanding, but requiring depth sensors or complete camera measurements would exclude some robot data. Whether training can use unevenly annotated data while still building spatial understanding changes what a robot training recipe can draw on.
SLA aligns a robot model’s visual features with 3D spatial signals during continued training and task-specific training, aiming to make those features more useful for manipulation.
SLA starts from the pretrained π0.5 policy and combines action learning with alignment to spatial features from a frozen 3D model, during continual pre-training and downstream post-training. Its RoboTwin evaluation used 50 clean demonstrations per task, then tested 50 tasks under Clean2Random scene variation.
SLA’s task-averaged complete-task success rate was 72.3%, versus 46.0% for π0.5. The result is consistent with the full training approach helping in varied scenes, though the comparison does not isolate alignment alone.
The 72.3% success rate is averaged across 50 RoboTwin tasks in Clean2Random. The selected passages do not fully specify the π0.5 baseline’s starting checkpoint or post-training procedure, and the result does not show that SLA leads on every task or test.
What the evidence shows
- A spatial training method
The Spatial-Language-Action Model (SLA) is a robotic foundation-model training method. It addresses a gap in policies trained from 2D images: precise manipulation calls for spatial reasoning, but robot datasets have uneven depth and camera annotations.
SLA uses 3D information as training guidance, aiming to improve visual features without adding required depth or calibration inputs at use time.
- The representation gap
A depth probe asks whether a model’s frozen visual features already carry spatial structure: researchers train a separate head to predict depth from those features. Features learned from 2D images without alignment did not yield meaningful spatial structure, while aligned features performed better in the probe. This supports a representation-level claim, not a direct claim about successful manipulation.
- A gain under scene variation
The Clean2Random comparison indicates that the full SLA training approach performs better under scene variation beyond clean training conditions.
Because the compared approaches differ in training, this result does not isolate spatial alignment as the sole cause or establish a lead on every task or test.
- Both stages matter here
A separate RoboTwin comparison varied whether spatial alignment was used during continual pre-training, post-training, both, or neither. With the full pre-training data, the both-stage setup scored 72.3%, versus 63.2% with alignment only during continual pre-training.
The result supports using alignment in both tested stages in this comparison, while the broader result remains a comparison of training configurations rather than an isolated explanation of every gain.
- A boundary on unseen tasks
The evidence for tasks without post-training demonstrations is limited in RoboDojo-Sim’s Open group: seven of its eight tasks were absent from those demonstrations, and only one had any successful episodes.
That boundary sits alongside evaluations across several simulated and real-robot settings, but those reported settings do not establish performance beyond the tested conditions.
Deep
- Original paper authors
- Fuhao Li, Wenxuan Song, Qizhou Huang, Ziyang Zhou, Jingbo Wang
- Submitted to arXiv
- License
- arXiv non-exclusive