Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction

2026-06-23

AI Summary

This work addresses the limitations of existing vision token pruning methods, which rely on text–vision attention scores that are often biased by the model’s inherent priors, leading to the erroneous removal of task-relevant tokens. To overcome this, the authors propose PriorTR, a training-free token compression approach that explicitly disentangles model priors from task-conditioned attention in a single forward pass by introducing a null instruction token. PriorTR leverages this null token as a probe and employs a prior–posterior contrastive metric to more accurately assess token importance, enabling efficient and precise pruning. Experiments demonstrate that PriorTR significantly outperforms current training-free baselines across multiple multimodal benchmarks and multimodal large language models, maintaining superior accuracy–efficiency trade-offs even under aggressive compression ratios.

Abstract

Visual token reduction has emerged as an effective strategy for accelerating Multimodal Large Language Models (MLLMs). Many existing methods prune tokens by ranking text-visual attention scores. However, we show that attention is often dominated by a model-induced prior: even without textual instruction, MLLMs tend to focus on certain task-agnostic regions. Consequently, the attention scores of instruction-conditioned tokens are suppressed, increasing the risk that these tokens are discarded during pruning. To address this issue, we propose Prior-Corrected Token Reduction (PriorTR), a training-free token reduction method that explicitly separates task-conditioned attention from the model-induced prior. PriorTR estimates the attention map of the prior, and contrasts it with the task-conditioned attention distribution to measure the additional usable information contributed by each visual token. Importantly, PriorTR computes both the model-induced prior and the task-conditioned posterior within a single forward pass by introducing a null token that serves as an instruction-agnostic probe in the attention block. This design avoids duplicated propagation. Extensive experiments across multiple multimodal benchmarks and MLLMs demonstrate that PriorTR consistently improves the trade-off between accuracy and efficiency over strong training-free baselines, particularly under aggressive token budgets.

Problem

• Multimodal Large Language Models
• Visual Token Reduction
• Attention Prior
• Task-Conditioned Attention
• Model-Induced Prior

Innovation

• Prior-Corrected Token Reduction
• Multimodal Large Language Models
• Attention Prior
• Training-Free Acceleration
• Visual Token Pruning