PLPHP: Per-Layer Per-Head Vision Token Pruning for Efficient Large Vision-Language Models

2025-02-20

AI Summary

To address the low decoding efficiency and high memory overhead in Large Vision-Language Models (LVLMs) caused by redundant visual tokens, this paper proposes a hierarchical, head-wise fine-grained visual token pruning method. Our approach innovatively introduces inter-layer dynamic retention rate allocation and head-level independent pruning, leveraging the re-attention phenomenon of visual tokens in self-attention to enable differentiated token removal. We further integrate KV cache compression to enhance inference efficiency. Evaluated on standard multimodal benchmarks, our method maintains strong multimodal understanding capability while achieving an 18% speedup in decoding, reducing KV cache size by over 50%, and incurring only a marginal performance drop of 0.46% in overall accuracy. Notably, it even yields slight improvements on multi-image tasks. These results demonstrate significant gains in both inference efficiency and deployment feasibility for LVLMs.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across a range of multimodal tasks. However, their inference efficiency is constrained by the large number of visual tokens processed during decoding. To address this challenge, we propose Per-Layer Per-Head Vision Token Pruning (PLPHP), a two-level fine-grained pruning method including Layer-Level Retention Rate Allocation and Head-Level Vision Token Pruning. Motivated by the Vision Token Re-attention phenomenon across decoder layers, we dynamically adjust token retention rates layer by layer. Layers that exhibit stronger attention to visual information preserve more vision tokens, while layers with lower vision attention are aggressively pruned. Furthermore, PLPHP applies pruning at the attention head level, enabling different heads within the same layer to independently retain critical context. Experiments on multiple benchmarks demonstrate that PLPHP delivers an 18% faster decoding speed and reduces the Key-Value Cache (KV Cache) size by over 50%, all at the cost of 0.46% average performance drop, while also achieving notable performance improvements in multi-image tasks. These results highlight the effectiveness of fine-grained token pruning and contribute to advancing the efficiency and scalability of LVLMs. Our source code will be made publicly available.

Problem

• Enhance LVLMs inference efficiency
• Reduce visual tokens in decoding
• Implement fine-grained pruning method

Innovation

• Dynamic token retention rates
• Head-level independent pruning
• Layer-level rate allocation