Problem
Research questions and friction points this paper is trying to address.
vision-language models
visual token pruning
efficient inference
token importance
evidence construction
DIVE: Dynamic Iterative Visual Evidence Construction for Efficient Vision-Language Models
🤖 AI Summary
This work addresses the inefficiency of vision-language models caused by excessively long visual token sequences, a challenge exacerbated by existing single-pass scoring pruning methods that struggle to accurately assess token relevance within contextual interactions. To overcome this limitation, the authors propose a training-free, dynamic iterative pruning framework that reframes token pruning as a process of constructing complementary visual evidence. The approach leverages residual conditional scoring, iterative token selection, and joint updates of visual and prompt residuals to transcend the constraints of one-shot evaluation. Evaluated across eight image understanding benchmarks, the method retains only 11.1% of visual tokens while preserving 98.2% of the original model’s average performance, substantially improving inference efficiency.
📝 Abstract
Visual inputs in vision-language models (VLMs) are often encoded into substantially longer token sequences than text, making visual tokens a major bottleneck for efficient inference. Abundant recent methods address this bottleneck by scoring token importance and pruning low-scoring tokens in a single pass. However, one-shot scoring is insufficient because a token's prompt-relevant usefulness depends on the evidence already retained. Motivated by this insight, we introduce DIVE (Dynamic Iterative Visual Evidence Construction), a training-free framework that recasts visual-token pruning as dynamic evidence construction. DIVE repeatedly selects the remaining token with the highest residual-conditioned score, updates the visual and prompt residuals to discount the evidence already explained, and re-evaluates the remaining tokens. This select-update-re-evaluate process builds a retained set of complementary, prompt-relevant evidence. Experiments across eight image-understanding benchmarks show that DIVE consistently preserves performance across token budgets. With an 88.9% reduction in visual tokens, DIVE retains 98.2% of the uncompressed model's average performance. Code is available at https://github.com/Zhong-Chenchen/DIVE.git.
Innovation
Methods, ideas, or system contributions that make the work stand out.
dynamic iterative pruning
visual evidence construction
vision-language models
token reduction
training-free framework
🔎 Similar Papers
No similar papers found.
