Robots can mix token reuse and pruning to speed up vision-language control
TRIAGE: Efficient Vision-Language-Action Inference Needs Both Reuse and Pruning · arXiv:2610.34519
In plain words
A robot can spend a long time processing camera images before choosing an action. Skipping image details saves time, but throwing away useful details or reusing outdated ones can lead to worse actions.
The paper introduces TRIAGE, which decides for each visual detail whether to recompute it, reuse an earlier computation, or drop it. In the reported robot-task tests, TRIAGE kept average success close to recomputing every detail while reducing input-processing time on a tested robot computer.
The research
A robot’s vision-language model must process many visual tokens to choose actions, which can slow onboard inference. Skipping tokens can save time, but pruning may discard useful information and reusing an old computation may give the model stale information. How should it choose what to recompute and what to do with the rest?
TRIAGE assigns each visual token to recomputation, cache reuse or pruning, using importance and cache validity to spend a fixed computation budget.
The authors test whether a visual token’s drift can show when reusing its cached state is better than pruning it. In a token-level diagnostic, low-drift reuse was no worse than pruning, while the harmful-drift threshold differed between the two tested backbones.
TRIAGE uses that distinction alongside token importance to allocate a fixed recomputation budget. In reported benchmark averages, it kept success within 3 percentage points of full recomputation while reducing input-processing latency on a Jetson AGX Orin.
The evaluations cover two vision-language-action backbones and the reported LIBERO and SimplerEnv settings. TRIAGE’s settings were selected empirically, and the authors say the drift threshold may need recalibration for a different backbone, task or scene speed.
What the evidence shows
- The cost of visual input
TRIAGE is a training-free inference method for vision-language-action models, which use camera images and language instructions to produce robot actions. The paper addresses a specific speed problem: visual tokens make up most of the input, and processing them at every control step can slow inference on an onboard robot computer.
Skipping tokens saves computation, but the model must decide what to do with each token it does not recompute.
- When old computations help
The authors ask whether a token’s change since its last computation can indicate whether its cached state remains useful. They compare reuse and pruning on the same probed tokens, with all other tokens recomputed, and judge both against actions from full recomputation.
In this diagnostic, low-drift reuse was no worse than pruning, while high-drift reuse could produce substantially more action error. The drift threshold for that change differed between the two tested backbones.
- How TRIAGE assigns tokens
TRIAGE turns that distinction into three choices. It uses importance to spend a fixed recomputation budget on important tokens whose cached states are invalid.
For the other tokens, it reuses a cache when the token’s drift is below the threshold and the cache is not too old; otherwise it prunes the token. The authors’ comparison of same-budget variants on LIBERO-Object, a robot manipulation test measuring episode success, shows the value of combining reuse and pruning in that setting.
- Evidence and its boundaries
At 18 recomputed tokens per query on LIBERO-Object, TRIAGE succeeded in 75.8% of 500 episodes, versus 22.4% when all remaining tokens were pruned and 0.0% when all were reused. In the broader reported averages, TRIAGE stayed within 3 percentage points of full recomputation in success rate while reducing input-processing latency by up to 48% on a Jetson AGX Orin.
On SimplerEnv, the authors also carried settings over without retuning and found average success close to full recomputation.
Deep
- Original paper authors
- Sooyoung Kwon, Carlee Joe-Wong, Kyunghan Lee
- Submitted to arXiv
- License
- arXiv non-exclusive