ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruning

2026-01-25

AI Summary

This work addresses the high computational cost of large vision-language models caused by redundant visual tokens, a challenge exacerbated by existing pruning methods that often discard critical information. To this end, we propose ViTCoP, a vision-text semantic collaborative pruning framework that filters redundant tokens in the vision encoder while leveraging the hierarchical structure of large language models to perform progressive co-pruning, thereby preserving informative and diverse visual tokens. We introduce, for the first time, the K-vector L2 norm as a token saliency metric, which is compatible with FlashAttention for efficient inference. Experiments demonstrate that ViTCoP achieves state-of-the-art performance on both image and video understanding tasks, significantly reducing inference latency and GPU memory consumption—particularly under extreme pruning ratios.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Security and Privacy: Privacy-enhancing technologies

Abstract

Large Vision-Language Models (LVLMs) incur high computational costs due to significant redundancy in their visual tokens. To effectively reduce this cost, researchers have proposed various visual token pruning methods. However, existing methods are generally limited, either losing critical visual information prematurely due to pruning in the vision encoder, or leading to information redundancy among the selected tokens due to pruning in the Large Language Models (LLMs). To address these challenges, we propose a Visual and Textual Semantic Collaborative Pruning framework (ViTCoP) that combines redundancy filtering in the vision encoder with step-wise co-pruning within the LLM based on its hierarchical characteristics, to efficiently preserve critical and informationally diverse visual tokens. Meanwhile, to ensure compatibility with acceleration techniques like FlashAttention, we introduce the L2 norm of K-vectors as the token saliency metric in the LLM. Extensive experiments on various Large Vision-Language Models demonstrate that ViTCoP not only achieves state-of-the-art performance surpassing existing methods on both image and video understanding tasks, but also significantly reduces model inference latency and GPU memory consumption. Notably, its performance advantage over other methods becomes even more pronounced under extreme pruning rates.

Problem

• Large Vision-Language Models
• visual token pruning
• computational redundancy
• information preservation
• model acceleration

Innovation

• visual-textual collaborative pruning
• token redundancy reduction
• hierarchical co-pruning
• L2 norm saliency
• large vision-language models