Problem
Research questions and friction points this paper is trying to address.
Vision-Language Models
visual token sequences
prefill costs
cross-modal alignment
attention quality
HAP: Head-Adaptive Visual Token Pruning via Cross-Modal Alignment
🤖 AI Summary
为解决高分辨率图像在视觉-语言模型中产生的高昂预填充成本问题，提出基于PAQ的视觉令牌修剪方法，通过跨模态对齐更精准地保留重要信息。
📝 Abstract
Recent Vision-Language Models encode high-resolution images into long visual token sequences, incurring prohibitive prefill costs. To compress them, existing methods score each visual token by averaging text-to-visual attention uniformly across all heads, which assumes every head matches the query. However, our empirical analysis shows that misaligned heads dominate the average, amplifying background tokens and drowning out fine-grained cues.
To address this, we propose PAQ (Prompt-Grounded Attention Quality), a metric quantifying how well each head aligns the prompt with image regions. Built on PAQ, our pruning proceeds in three stages. Given a target FLOPs budget, we first partition the transformer layers into groups and allocate a visual token budget to each. Within each group, we then aggregate per-head attention maps via PAQ-weighted softmax into a group-level matrix. Finally, we score visual tokens by this matrix's magnitude and retain the allocated budget per group. By weighting heads with PAQ, our method scores tokens by attention signals that more faithfully reflect prompt relevance, rather than diluting them through uniform averaging.
Across 18 benchmarks, our method delivers state-of-the-art trade-offs. Specifically, on LLaVA-1.5-7B (9 tasks), retaining only \textbf{5.6\%} tokens preserves \textbf{99.1\%} of the original performance, surpassing the strongest baseline AutoPrune by 4.2 points. Code is available in https://github.com/baokou-fw2/HAP.
Innovation
Methods, ideas, or system contributions that make the work stand out.
PAQ
visual token pruning
cross-modal alignment
attention quality
🔎 Similar Papers
No similar papers found.
