TouchScale pairs 500 hours of human video with touch
In plain words
TouchScale is a 500-hour dataset that records what people see and what their hands feel during everyday activities and object manipulation. Cameras show hand movement, while wearable gloves record pressure on both hands; the recordings are lined up in time.
Models trained on TouchScale predicted contact better on a test using a tactile sensor they had not seen in training than a model trained on a similarly-sized existing dataset. The paper also reports gains in its tested action-recognition and robot tasks.
The research
Cameras show hand movement but leave out where contact happens and how much pressure a hand applies. Can a large, consistently collected record of both sight and touch improve predictions on an unfamiliar tactile sensor, and can it also help visual learning and robot manipulation?
TouchScale is a 500-hour dataset aligning human camera views with hand pressure, enabling tests of touch prediction, visual learning, and robot training.
The authors trained the same tactile-prediction architecture on similarly-sized TouchScale and EgoTouch data, then tested both on EgoTactile, a sensor absent from training. The TouchScale-trained model scored higher on contact overlap.
With the model and test protocol fixed, increasing TouchScale data from 10% to the full dataset also raised that score, supporting transfer and a data-scale effect in this test. TouchScale pretraining also produced the highest average action-recognition accuracy among the compared visual-and-touch datasets on three benchmarks, both with a trained classifier and when the full model was tuned for recognition.
In four robot tasks, average success was higher after TouchScale mid-training than without it, with the same robot demonstrations used for training.
The tactile transfer result is limited to EgoTactile and does not establish transfer to every unseen sensor. Robot tests cover one platform and four tasks, and the role of future-tactile prediction in robot control remains open.
Analysis
Deep
- Authors
- Dayou Li, Hao Wang, Qianqian Yang, Zihao Zhu, Haoquan Fang
- License
- arXiv non-exclusive