In plain words
TAPESTRY is a robot-control method that reuses a model trained to generate video. It turns each robot action step into a patch-like piece the video model already knows how to process, then trains the same model on videos and robot demonstrations.
When used, it takes a camera image and a language goal and directly outputs a short sequence of actions, without generating video.
In four simulated task groups whose evaluation examples had video but no action labels, its average task success was 76%, compared with 70% for each of two video-based methods.
The research
Video models can learn visual patterns of physical interaction, but a robot controller that generates future video during control pays the cost of making images it does not need to show. Can a robot use those pretrained visual dynamics directly to choose its next actions?
TAPESTRY shares a pretrained video model between video and action training, letting it turn a camera image and language goal into action chunks without generating video.
On LIBERO, a simulation test of robot task completion, action labels were available for one suite and four evaluation suites had video only. TAPESTRY averaged 76% completed rollouts across those suites, against 70% each for Cosmos Policy and DreamZero.
This supports action generation from a video-pretrained model in this setting, though the comparison cannot isolate which design choices explain the gap.
The four evaluation suites had videos but no action labels, so these rates are not directly comparable with standard fully supervised LIBERO scores.
TAPESTRY and its comparison methods also differed in architecture, training objectives, duration, and inference procedure, so the score gap does not identify which design choice explains it.
What the evidence shows
- One backbone, two modalities
TAPESTRY, short for Tokenizing Actions as Video Patches in a Shared Transformer Policy, is a robot-control method. It addresses a transfer tradeoff: generating predicted video during control is costly, while a separate action module trained from scratch cannot directly use the pretrained video model’s weights. TAPESTRY instead routes robot actions through that model.
- From actions to tokens
Each robot action step is projected and normalized into a patch-sized unit, then passed through the same video patchifying layer and transformer. Training shares that backbone across video denoising, video-and-action denoising on demonstrations, and action-only denoising conditioned on the clean camera image and language goal. At use, only action tokens are produced, in a single-step from noise.
- Scores on video-only tasks
LIBERO is a simulation test of robot manipulation task completion. Action labels were available only for LIBERO-90; four other suites supplied video but no action labels.
The paper’s rollout evaluation reports an average task-success rate of 76% for TAPESTRY and 70% each for Cosmos Policy and DreamZero, over 100 attempts per suite. The paper does not name who judged completion.
- Diagnosing rough action output
To explain rough action output, the authors measured jitter across noise levels on 50 held-out action chunks. Inputs were formed by adding noise to true actions, so they stayed within the training distribution.
Mean jitter rose from 0.92 to 1.71 times the ground-truth jitter band as noise fell. In a separate comparison of signals from the same episodes, f95 was 0.352 Hz for actions and 1.992 Hz for video pixels. The authors interpret this frequency gap as a source of jitter.
- Tests on a real robot
On a real bimanual YAM robot, TAPESTRY completed 28 of 30 attempts across three trained tasks, compared with 15 of 30 for a from-scratch method. In three changed conditions with novel placements, motions, or object combinations, it completed 10 of 15 full tasks versus DreamZero’s 4 of 15.
The authors’ separate reported overall margin over DreamZero was not statistically significant (27±11%, p=0.07); each condition had only five attempts. In the object-composition condition, TAPESTRY completed all five attempts, while each other method completed none.
- Shorter inference time
The timing test compared action-chunk generation on the same RTX 5090 GPU. TAPESTRY’s 2B model produced a 16-action chunk in 84 milliseconds; the 5B DreamZero took 3.7 seconds after added caching and activation checkpointing.
The paper reports a roughly 44-fold timing difference, but the models and inference procedures differ, so the result does not isolate one design choice.
Deep
- Original paper authors
- Jeremy A. Collins, Seungjae Lee, Rachanon Wachakorn, Krishnan Srinivasan, Animesh Garg
- Submitted to arXiv
- License
- arXiv non-exclusive