Beyond Intermediate States: Explaining Visual Redundancy through Language

2025-03-26

AI Summary

Visual token redundancy in multimodal large language models (MLLMs) lacks a rigorous definition, and existing pruning strategies lack semantic grounding. Method: We propose the first language-feedback-based visual redundancy quantification framework: (1) identifying redundant visual prototypes under context-free conditions; (2) modeling token-level and context-level impacts on language generation via visual token perturbation and corresponding textual output variation analysis; and (3) integrating ViT-[cls] correlation analysis with text-to-image attention modeling. Contribution/Results: We discover that low-correlation visual tokens may still encode critical semantic information. Experiments across single-image, multi-image, and video understanding tasks show that pruning 80–90% of visual tokens preserves 90–110% of original performance, substantially improving inference efficiency and context utilization while maintaining task fidelity.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multi-modal Large Langue Models (MLLMs) often process thousands of visual tokens, which consume a significant portion of the context window and impose a substantial computational burden. Prior work has empirically explored visual token pruning methods based on MLLMs' intermediate states (e.g., attention scores). However, they have limitations in precisely defining visual redundancy due to their inability to capture the influence of visual tokens on MLLMs' visual understanding (i.e., the predicted probabilities for textual token candidates). To address this issue, we manipulate the visual input and investigate variations in the textual output from both token-centric and context-centric perspectives, achieving intuitive and comprehensive analysis. Experimental results reveal that visual tokens with low ViT-[cls] association and low text-to-image attention scores can contain recognizable information and significantly contribute to images' overall information. To develop a more reliable method for identifying and pruning redundant visual tokens, we integrate these two perspectives and introduce a context-independent condition to identify redundant prototypes from training images, which probes the redundancy of each visual token during inference. Extensive experiments on single-image, multi-image and video comprehension tasks demonstrate the effectiveness of our method, notably achieving 90% to 110% of the performance while pruning 80% to 90% of visual tokens.

Problem

• Reducing computational burden of MLLMs by pruning visual tokens
• Overcoming limitations of prior visual redundancy definition methods
• Developing reliable method to identify and prune redundant visual tokens

Innovation

• Integrates ViT-cls association and attention scores
• Introduces context-independent redundancy identification condition
• Prunes 80-90% visual tokens with minimal performance loss