LapMerge makes ordered token merging learnable
En termes simples
Long token sequences can be costly to process, but simply dropping tokens may lose useful information. LapMerge is a method that turns an ordered sequence into an exact, smaller number of tokens by blending nearby tokens according to learned weights.
In questions about objects in images, it was compared with a method that kept selected visual tokens. LapMerge scored higher at the tighter token budget, while the difference at a less tight budget was uncertain. Its preference for nearby tokens may limit it when useful information is far apart.
La recherche
When a model processes a long ordered sequence, reducing its tokens can lower later processing cost, but hard score-based cuts can make the scorer impossible to train through. Can a method produce an exact smaller sequence while keeping the token weights connected to the result by a learning signal?
LapMerge smooths token positions and blends inputs within equal-mass intervals, producing an exact output budget with differentiable, linear-work merging.
LapMerge addresses a basic compression question: can an ordered sequence be reduced to an exact number of tokens while still allowing token scores to learn? Hard cumulative-weight cuts can jump when scores change, blocking the gradient signal to the scoring component.
The method smooths token positions, divides their mass into equal portions, and blends contributing tokens. On POPE, a test of whether a vision-language model answers object-presence questions correctly, the frozen model scored higher than selection at a 10% visual-token budget. At 30%, the paired uncertainty interval included zero, so the evidence for an advantage is weaker.
These results do not show that LapMerge improves accuracy across all tasks or token orders. Its aggregation favors nearby positions, and the POPE advantage over selection was not clearly separated from zero at the 30% visual-token budget.
Analyse
Détails
- Licence
- CC BY 4.0