Human motion helps humanoid robots follow instructions
In plain words
Teaching a humanoid robot to follow instructions is hard: its legs, body, arms and fingers must work together, and there are few robot demonstrations for varied tasks.
VioLA is a control policy that predicts desired body and hand movements; pretrained robot controllers turn those movements into joint commands.
The researchers also used annotated human movement to train VioLA. On one robot, it completed all trials on six walking and posture tasks without task-specific training. It did not complete every manipulation task, and the tests covered one robot in one laboratory.
The research
Can human movement recordings help a humanoid follow whole-body instructions when robot demonstrations are scarce? Human motion does not contain robot commands, and a humanoid must coordinate many joints while keeping its balance. VioLA tests whether both sources can teach the same movement targets.
VioLA predicts shared body and hand movements from human and robot demonstrations, letting pretrained controllers execute instructions on a humanoid.
VioLA predicts body and hand movements that pretrained robot controllers can execute. Human and robot demonstrations use the same movement representations, so annotated human motion can train the policy without a robot providing each demonstration.
On one Unitree G1, VioLA completed all locomotion trials and 31 of 35 manipulation trials without task-specific training.
The comparison with released policies shows strong completion on these tested tasks, while comparisons of training schedules show that human data helped locomotion but not manipulation.
These results come from five trials per task on one Unitree G1 in one laboratory. The task families overlap training, although specific episodes, the laboratory environment and object instances were held out. VioLA also failed all five trials of a separately analyzed bottle-carrying task.
Analysis
Deep
- License
- arXiv non-exclusive