A shared touch model improves robot task success
Learning Heterogeneous Tactile Representation for General Dexterous Manipulation · arXiv:2611.12806
In plain words
Different touch sensors describe contact in different ways, so a model trained for one sensor may not understand another.
LingBot-Touch is a model that turns readings from three sensor types into a shared description of fingertip contact, then uses that description to help answer questions about contact or guide a robot.
On 26 real-robot tasks, adding its frozen features raised average success for each of four tested robot controllers compared with giving them no touch input. The average does not mean every controller-task pairing improved.
The research
A robot can receive a lot of touch data and still fail to use it well if each sensor describes contact differently.
These results change the case for adding touch to manipulation: across the tested tasks, a learned shared description helped more on average than raw sensor readings, while leaving task-by-task variation visible.
LingBot-Touch learns shared contact features from three tactile sensor types, then supplies frozen features for contact probes and robot manipulation.
Researchers trained LingBot-Touch on recordings from three sensor types, then added its frozen features to four robot controllers and compared them with versions receiving no touch or raw tactile signals.
On 26 real-robot tasks, the feature-based versions had higher mean success than the no-touch versions; the raw-signal versions gained less on average and sometimes scored lower.
This supports using learned contact features in these tested tasks, while the raw-input comparison suggests that adding sensor readings alone did not produce the same average gains.
The robot results average equally across 26 tasks, so they do not mean every policy improved on every task. The tests covered two robot hands, three sensing principles in pretraining, and fingertip-focused contact.
What the evidence shows
- One description across sensors
LingBot-Touch is a tactile foundation model that learns a shared description of contact from three kinds of sensors. This addresses a practical mismatch: the sensors use different layouts, channels, resolutions, and sampling rates, even when they record the same physical contact. Its purpose is to make tactile features reusable for contact understanding and robot manipulation.
- Test the learning objective
Compared with signal reconstruction, latent prediction produced more useful frozen features on these contact probes at both tested data sizes. This suggests the objective matters for the representations LingBot-Touch learns under these matched training conditions.
- Check transfer to XHand1
The shared features also worked on XHand1, a magnetic sensor model not included in pretraining. LingBot-Touch used a frozen encoder and task-specific probe without sensor-specific parameter updates; FTP-1 used a trained XHand1-specific input adapter and probe.
LingBot-Touch performed better on all five reported contact probes. This comparison supports transfer to XHand1 under these conditions, while preserving the difference in sensor-specific adaptation.
- Features help robot tasks
The comparison indicates that learned tactile features were more useful on average than adding raw readings in this robot test. Raw readings helped less overall and sometimes reduced success, so the result does not imply that touch input alone reliably helps every task.
- Where the evidence stops
The result is evidence for a reusable, fingertip-centered representation within the tested sensors and hands. The paper notes that capacitive sensors were absent from pretraining, contact on palms and other finger regions was underrepresented, and experiments need a wider range of hand embodiments.
These limits matter because a shared interface learned from three sensing principles and fingertip-focused layouts may not capture all tactile contact.
Deep
- Original paper authors
- Bing Han, Yichuan Peng, Haoyu Dong, Yuanmeng Hu, Yongyi Jia
- Submitted to arXiv
- License
- CC BY 4.0