SheafPrune ranks visual tokens before language decoding
In plain words
When an image question is answered, some visual details matter only because of the question or because they connect to other details. SheafPrune is a method that ranks image tokens and removes some before the language model processes them.
It scores tokens using the question and links among visual details, then adjusts for how many neighbors each token has. Across four tested models and seven pruning budgets, it had the highest average retained test scores in 22 of 28 comparisons, though other methods led in some cases.
The research
If a model must discard many image tokens before answering, which details should it keep? A token's value may depend on the question or on its relationships with other visual tokens, so the paper asks whether accounting for those links can improve selection under a fixed budget.
SheafPrune combines question-to-image relevance with constrained links among visual tokens and adjusts for neighbor counts to rank a fixed-size subset.
The selector compares text and visual features, combines instruction relevance with a visual prior, then passes scores along selected visual-token links. It adjusts scores for neighbor counts and removes unselected tokens before the language-model decoder.
Across four models and seven budgets, researchers found its average fraction of unpruned scores retained across ten tests was highest in 22 of 28 comparisons, with correction settings chosen separately for each comparison. This supports the method across the tested grid, with exceptions at the tightest budget.
The main comparison reports the best correction setting separately for each model and budget, and the paper does not establish that these choices used a held-out validation set.
Its aggregate scores do not establish statistical significance, and the grid-coverage check does not explain the accuracy exceptions.
Analysis
Deep
- Original paper authors
- Xiaoyang Ma, Chi Wang, Hong Sun
- Submitted to arXiv
- License
- CC BY 4.0