One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs

2026-06-12

AI Summary

This work addresses the performance degradation in existing large vision-language models caused by static visual token pruning, which permanently discards information inaccessible to subsequent layers. To overcome this limitation, the authors propose an adaptive, hierarchical visual token selection mechanism that dynamically determines, via lightweight selectors, which tokens to process or skip at each layer, followed by cross-layer token fusion—enabling full-model adaptive compression without retraining. Key innovations include layer-wise differentiated token retention and reuse, a low-rank attention approximation guided by importance consistency constraints, and a multi-stream token routing and fusion strategy. Experiments on LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL demonstrate that retaining only 11% of the original visual tokens achieves 96.7% of the baseline model’s accuracy.

Abstract

Large Vision-Language Models (LVLMs) have achieved remarkable success across diverse multimodal tasks, yet their practical deployment remains constrained by the computational burden arising from lengthy visual tokens. While visual token pruning has emerged as a promising solution, existing methods suffer from a fundamental limitation: once tokens are pruned at a specific layer, they become inaccessible to all subsequent layers, leading to premature information loss that can compromise model performance. Through empirical studies, we observe that different layers exhibit distinct visual region focus, indicating a varying optimal token subset across layers. Motivated by this insight, we propose Adaptive Layer-wise Visual Token Selection (ALVTS), a novel framework that breaks away from the conventional static token pruning paradigm. ALVTS incorporates a lightweight token selector to identify and route important tokens for further processing, while allowing less important tokens to skip the layer, thus minimizing computational redundancy. These two streams of tokens are seamlessly reintegrated before being fed into subsequent layers, facilitating adaptive compression across the entire model. Grounded in our importance consistency constrained low-rank approximation, the proposed token selection module closely emulates the full attention mechanism, effectively capturing its essential patterns without requiring model retraining. Extensive experiments on LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL validate the effectiveness of our method. With an 89% token compression ratio, ALVTS retains 96.7% of the original model's accuracy, achieving a superior efficiency-accuracy trade-off for LVLM inference.

Problem

• visual token pruning
• Large Vision-Language Models
• computational burden
• premature information loss
• layer-wise token selection

Innovation

• Adaptive Layer-wise Token Selection
• Visual Token Pruning
• Large Vision-Language Models
• Efficient Inference
• Low-Rank Approximation