One visual encoder links scene state, language, and image detail
In plain words
A world model predicts how a scene will change when a robot acts. The paper introduces ALPS, an image encoder for world models that aims to show where objects are, connect images to language, and preserve details that can be reconstructed as pixels. It learns from still images and text, then practices matching features across altered views.
In simulated object-manipulation tests, a probe using ALPS features estimated object positions with lower average error than probes using other tested ViT-B encoders. A larger encoder matched its average.
The research
World models predict how a scene changes after actions, so their visual features need to capture object locations and states.
Can one frozen encoder also connect those scenes to language and preserve enough detail for pixel reconstruction, rather than relying on representations specialized for only part of the task?
ALPS trains shared image features for object state, language, and pixel reconstruction, then refines them across views for world models.
Can one frozen image encoder represent object state, connect scenes to language, and retain details for reconstruction? Existing encoders often cover only some of these abilities, making it difficult for world models to use one representation across tasks.
ALPS trains shared image features for language, reconstruction, and spatial relationships, then refines them across altered views. In simulated LIBERO object-manipulation tests, a probe using its frozen features had lower average position error than probes using other tested ViT-B encoders. A separate policy evaluation also found strong control results.
These results come from a ViT-B/16 encoder and simulated tasks, with control and world-model planning tested separately. They do not show how ALPS would perform on a real robot or in one system combining language, planning, and image reconstruction.
Analysis
Deep
- Original paper authors
- Yanqing Liu, Letian Zhang, Chengzhi Liu, Zijun Wang, Jieru Mei
- Submitted to arXiv
- License
- arXiv non-exclusive