Not All Attention Heads Contribute to Critical Visual Token Selection: Head-Aware Pruning Matters More

2026-08-25

AI Summary

为解决视觉语言模型中视觉标记过多导致的效率问题，提出ProViP框架，通过基于嵌入相似性和头感知修剪方法逐步移除冗余视觉标记，提高任务性能和推理效率。

Abstract

Vision-Language Models (VLMs) have exhibited impressive performance across diverse visual scenarios. However, this success comes at the cost of explosive growth in visual tokens, which imposes substantial memory and computational overhead during inference, ultimately increasing latency. To improve VLM inference efficiency, a typical class of visual token pruning methods estimates token importance by aggregating attention scores across all heads in the pruning layer of the Large Language Model (LLM) backbone and prunes tokens based on aggregated scores. However, in this paper, we reveal a compelling phenomenon: the capability to pinpoint critical visual tokens is concentrated within a small fraction of heads. Aggregation exclusively on these heads can improve task performance. Inspired by this observation, we propose ProViP, a training-free progressive visual token pruning framework. ProViP first removes redundant visual tokens based on the embedding similarity of input tokens before reasoning of the LLM backbone, and then further prunes tokens during reasoning via head-aware pruning. Experiments demonstrate that ProViP delivers outstanding task performance and inference efficiency. For instance, when applied to LLaVA-1.5-7B, ProViP retains 95.9% of the original performance and achieves 1.62x inference speedup under an 88.9% pruning ratio.

Problem

• Vision-Language Models
• visual tokens
• attention heads
• pruning
• inference efficiency

Innovation

• head-aware pruning
• visual token pruning
• inference efficiency
• embedding similarity