Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization

2025-05-28

AI Summary

To address the excessive image tokens and prohibitive computational overhead induced by high-resolution image encoding in Large Vision-Language Models (LVLMs), this paper proposes a stage-wise balanced pruning strategy. Unlike existing methods relying solely on attention scores or local token diversity, our approach is the first to jointly model the synergistic impact of token pruning on both current-layer (local) and subsequent-layer (global) representations, enabling dynamic priority switching via a multi-stage calibration mechanism. Leveraging a small calibration dataset, the method evaluates tokens by integrating attention magnitude with feature diversity, and implements pruning through a plug-and-play lightweight architecture. Evaluated across multiple LVLMs, it achieves a 78% image token compression rate while preserving 96.7% of original task performance on average—substantially outperforming state-of-the-art pruning baselines.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Vision-Language Models (LVLMs) have shown impressive performance across multi-modal tasks by encoding images into thousands of tokens. However, the large number of image tokens results in significant computational overhead, and the use of dynamic high-resolution inputs further increases this burden. Previous approaches have attempted to reduce the number of image tokens through token pruning, typically by selecting tokens based on attention scores or image token diversity. Through empirical studies, we observe that existing methods often overlook the joint impact of pruning on both the current layer's output (local) and the outputs of subsequent layers (global), leading to suboptimal pruning decisions. To address this challenge, we propose Balanced Token Pruning (BTP), a plug-and-play method for pruning vision tokens. Specifically, our method utilizes a small calibration set to divide the pruning process into multiple stages. In the early stages, our method emphasizes the impact of pruning on subsequent layers, whereas in the deeper stages, the focus shifts toward preserving the consistency of local outputs. Extensive experiments across various LVLMs demonstrate the broad effectiveness of our approach on multiple benchmarks. Our method achieves a 78% compression rate while preserving 96.7% of the original models' performance on average.

Problem

• Reduces computational overhead in Vision-Language Models
• Improves token pruning considering local and global impacts
• Balances pruning stages for optimal performance preservation

Innovation

• Balanced Token Pruning for multi-stage optimization
• Calibration set divides pruning into stages
• Emphasizes global impact early, local consistency late