TileShare reduces repeated work in tiled vision decoders
TileShare: Training-Free Acceleration of Overlapping Tiled ViT Decoding · arXiv:2610.32233
In plain words
Some vision decoders are trained on small windows but later process larger images or videos by splitting them into overlapping tiles. That repeats calculations where tiles overlap, while simply removing tiles can change how each tile pays attention.
TileShare is a method that shares many calculations across tiles but keeps each tile’s attention.
In tests with MiniMax-H3, keeping the first six layers separate gave quality results comparable to the original on DAVIS videos, with a similar result on a separate set of UVG videos under the reported measures.
The research
When a fixed-window Vision Transformer decoder handles a larger input, overlapping tiles preserve each tile’s attention context but repeat calculations on shared areas. Can a decoder share that repeated work without materially changing the reconstructed image or video quality?
TileShare shares per-token calculations across overlapping tiles while preserving each tile’s attention, then tests how many early layers must remain separate.
TileShare computes token-level normalization and projections once, still runs attention separately for each tile, and blends attention results for overlapping tokens. Researchers compared MiniMax-H3 with the released decoder while varying how many early layers stayed separate per tile.
On All-38, mean PSNR difference was −0.290 dB with every layer shared and −0.003 dB with four layers kept separate. PSNR compares reconstructed pixels with a reference; higher values mean closer pixel agreement.
The −0.003 dB difference was within the paper’s equivalence margin. Comparisons using six separate layers also supported comparable quality on DAVIS videos and on UVG videos evaluated after settings were frozen.
The quality evidence is specific to the tested models and video sets. All-38 includes clips used while choosing the settings; on the later-tested UVG videos, TileShare-6 matched the released decoder in PSNR and DISTS and was non-inferior in LPIPS, but did not meet the study’s stricter equivalence margin for that measure.
What the evidence shows
- Repeated work in overlap areas
TileShare is a training-free method for accelerating overlapping tiled Vision Transformer decoding. These decoders use overlapping tiles to process inputs larger than their training windows, but tokens in overlap areas are computed more than once.
In MiniMax-H3 at the reported resolutions, that repeated work accounted for 44–51% of per-token computation on average across tokens in a frame.
- Sharing while keeping attention
The method shares calculations that operate on individual tokens while preserving each tile’s attention. For tokens covered by multiple tiles, it blends their attention outputs with the released weights.
That creates one shared state for later layers, where the original decoder keeps a separate state for each tile. The initial layers can remain separate per tile to limit the effect of that change.
- Finding the source of quality loss
Researchers used MiniMax-H3 comparisons to locate the quality cost. PSNR compares reconstructed pixels with a reference; higher values mean closer pixel agreement.
When the MLP alone was shared, mean PSNR changed by −0.022 dB relative to the released decoder; full sharing from the first layer changed it by −0.290 dB. The authors interpret this contrast as evidence that the larger cost arises when attention reads blended states, rather than from sharing the MLP alone.
- Choosing how many layers to keep
The number of initial layers kept per tile affected output quality. On All-38, which combines eight development clips and 30 DAVIS videos, mean PSNR difference was −0.290 dB when all layers were shared and −0.003 dB when four layers stayed separate.
The latter is within the paper’s ±0.05 dB equivalence margin. With six layers separate, results on DAVIS-30 and later-tested UVG videos supported comparable quality, with a qualification for UVG LPIPS.
- Testing other decoders
The researchers also tested TileShare on MAGI-1 and ViTok-v2, using different per-tile depths. It matched the reported quality measures on the tested MAGI-1 video sets, with a qualification for LPIPS on UVG.
For ViTok-v2, it matched PSNR and improved LPIPS and DISTS on the tested image sets. An audit of 14 configurations found that some did not need tiling and that the required depth varied among those that did.
Deep
- Original paper authors
- Xiangzhong Ye, Zhongshen Li, Shen You, Ziyang Meng, Liang Xu
- Submitted to arXiv
- License
- arXiv non-exclusive