MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs

2025-08-25

AI Summary

Vision-language models (VLMs) suffer from inefficient inference due to redundant visual tokens, and existing pruning methods predominantly rely on unimodal cues, lacking mechanisms for multimodal co-optimization. Method: We propose a multimodal coverage maximization principle that jointly models semantic coverage relationships between visual and textual tokens—introducing multimodal alignment into visual token selection for the first time. We design a VLM surrogate to enhance textual representations for guiding pruning and formulate token selection as a maximum coverage problem. Contribution/Results: Our method achieves 1.87× inference speedup while retaining 98.7% of original performance across multiple VLMs. On LLaVA-1.5-7B, it maintains 87.7% of baseline performance using only four visual tokens—significantly outperforming unimodal pruning baselines.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Vision-Language Models (VLMs) demonstrate impressive performance in understanding visual content with language instruction by converting visual input to vision tokens. However, redundancy in vision tokens results in the degenerated inference efficiency of VLMs. While many algorithms have been proposed to reduce the number of vision tokens, most of them apply only unimodal information (i.e., vision/text) for pruning and ignore the inherent multimodal property of vision-language tasks. Moreover, it lacks a generic criterion that can be applied to different modalities. To mitigate this limitation, in this work, we propose to leverage both vision and text tokens to select informative vision tokens by the criterion of coverage. We first formulate the subset selection problem as a maximum coverage problem. Afterward, a subset of vision tokens is optimized to cover the text tokens and the original set of vision tokens, simultaneously. Finally, a VLM agent can be adopted to further improve the quality of text tokens for guiding vision pruning. The proposed method MMTok is extensively evaluated on benchmark datasets with different VLMs. The comparison illustrates that vision and text information are complementary, and combining multimodal information can surpass the unimodal baseline with a clear margin. Moreover, under the maximum coverage criterion on the POPE dataset, our method achieves a 1.87x speedup while maintaining 98.7% of the original performance on LLaVA-NeXT-13B. Furthermore, with only four vision tokens, it still preserves 87.7% of the original performance on LLaVA-1.5-7B. These results highlight the effectiveness of coverage in token selection.

Problem

• Reduces vision token redundancy in VLMs for efficiency
• Selects informative vision tokens using multimodal coverage
• Maintains model performance while accelerating inference speed

Innovation

• Leverages multimodal vision and text tokens
• Formulates token selection as maximum coverage problem
• Uses VLM agent to improve text token quality