D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning

2025-12-22

AI Summary

Long visual sequences impose prohibitive computational overhead on multimodal large language models (MLLMs), and existing token pruning methods fail on fine-grained localization tasks: importance-based approaches suffer from positional bias, while diversity-based methods neglect spatial structure and prompt relevance. To address this, we propose a dynamic collaborative pruning framework featuring two novel mechanisms: (1) bias-corrected importance scoring—leveraging debiased attention weights to select semantically critical tokens—and (2) hybrid graph-based maximum independent set (MIS) pruning—constructing a graph that jointly encodes spatial proximity and semantic similarity to preserve structural integrity. Evaluated on LLaVA-1.5-7B, our method reduces FLOPs by 74.2% while retaining 99.2% of original performance. On InternVL-2.5-8B for fine-grained localization, it achieves 85.7% accuracy under 90% pruning—a 63.53% improvement over SOTA—and establishes the first high-fidelity, localization-aware pruning paradigm.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Processing long visual token sequences poses a significant computational burden on Multimodal Large Language Models (MLLMs). While token pruning offers a path to acceleration, we find that current methods, while adequate for general understanding, catastrophically fail on fine-grained localization tasks. We attribute this failure to the inherent flaws of the two prevailing strategies: importance-based methods suffer from a strong positional bias, an inherent model artifact that distracts from semantic content, while diversity-based methods exhibit structural blindness, disregarding the user's prompt and spatial redundancy. To address this, we introduce D2Pruner, a framework that rectifies these issues by uniquely combining debiased importance with a structural pruning mechanism. Our method first secures a core set of the most critical tokens as pivots based on a debiased attention score. It then performs a Maximal Independent Set (MIS) selection on the remaining tokens, which are modeled on a hybrid graph where edges signify spatial proximity and semantic similarity. This process iteratively preserves the most important and available token while removing its neighbors, ensuring that the supplementary tokens are chosen to maximize importance and diversity. Extensive experiments demonstrate that D2Pruner has exceptional efficiency and fidelity. Applied to LLaVA-1.5-7B for general understanding tasks, it reduces FLOPs by 74.2% while retaining 99.2% of its original performance. Furthermore, in challenging localization benchmarks with InternVL-2.5-8B, it maintains 85.7% performance at a 90% token reduction rate, marking a significant advancement with up to 63. 53% improvement over existing methods.

Problem

• Reduces computational burden of long visual tokens in MLLMs
• Addresses failure of current pruning in fine-grained localization tasks
• Corrects positional bias and structural blindness in token pruning

Innovation

• Debiased attention scores identify critical tokens
• Hybrid graph models spatial and semantic token relationships
• Maximal Independent Set selection maximizes importance and diversity