AdaTP: Attention-Debiased Token Pruning for Video Large Language Models

2025-05-26

AI Summary

Video LLMs incur excessive computational overhead due to massive visual token generation from dense frame sampling. Existing attention-score-based token pruning methods suffer from two inherent biases: global bias (preference for initial/final frames) and local bias (excessive spatial concentration across frames), degrading pruning quality. This paper presents the first systematic modeling and decoupling of these biases, proposing a training-free, plug-and-play attention debiasing token pruning framework. Our method comprises dynamic token pruning, global positional bias correction, inter-frame spatial consistency suppression, and FLOPs-driven adaptive compression. Evaluated on LLaVA-OneVision-7B, it achieves lossless performance recovery using only 27.3% of the original FLOPs and attains state-of-the-art results across multiple video understanding benchmarks.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Video Large Language Models (Video LLMs) have achieved remarkable results in video understanding tasks. However, they often suffer from heavy computational overhead due to the large number of visual tokens generated from multiple video frames. Existing visual token compression methods often rely on attention scores from language models as guidance. However, these scores exhibit inherent biases: global bias reflects a tendency to focus on the two ends of the visual token sequence, while local bias leads to an over-concentration on the same spatial positions across different frames. To address the issue of attention bias, we propose $ extbf{A}$ttention-$ extbf{D}$ebi$ extbf{a}$sed $ extbf{T}$oken $ extbf{P}$runing for Video Large Language Models ($ extbf{AdaTP}$), a novel token pruning pipeline for Video LLMs. AdaTP integrates two dedicated debiasing modules into the pipeline, targeting global attention bias and local attention bias, respectively. Without the need for additional training, our method significantly reduces the computational overhead of Video LLMs while retaining the performance of vanilla models. Extensive evaluation shows that AdaTP achieves state-of-the-art performance in various commonly used video understanding benchmarks. In particular, on LLaVA-OneVision-7B, AdaTP maintains performance without degradation while using only up to $27.3%$ FLOPs compared to the vanilla model. Our code will be released soon.

Problem

• Video LLMs suffer from high computational overhead due to excessive visual tokens
• Existing token compression methods exhibit global and local attention biases
• AdaTP reduces computational costs while maintaining model performance via debiasing

Innovation

• Debiases global and local attention biases
• Reduces computational overhead significantly
• Maintains performance without additional training