TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference

2026-06-25

AI Summary

This work addresses the inefficiency of multimodal large language models (MLLMs) caused by excessive visual tokens, a challenge exacerbated by existing pruning methods that struggle to simultaneously preserve task relevance, information coverage, and semantic diversity. From first principles, the authors formulate visual token pruning as an optimal retention set selection problem and, for the first time, derive these three desiderata from an information-theoretic perspective, establishing a principled theoretical framework. They propose TOPS, a training-free and model-agnostic module that employs a top-down information-theoretic analysis combined with task-aware measures of diversity and coverage to efficiently select informative tokens. Extensive experiments across seven MLLM backbones and fourteen benchmarks demonstrate that TOPS significantly outperforms prior methods—e.g., removing 77.8% of visual tokens in LLaVA-NeXT while maintaining or slightly improving performance.

Abstract

Multimodal large language models (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens, which introduces substantial computational overhead. Visual token pruning offers a natural solution, yet existing methods are imperfect: attention-based criteria tend to retain redundant tokens, while diversity-based criteria are often agnostic to user instructions. Even methods that combine multiple criteria still lack a principled formulation of the intrinsic objective of token pruning. In this paper, we revisit visual token pruning from a first-principles perspective and formulate it as constructing Token Optimal Preservation Sets. Through a top-down information-theoretic analysis, we identify three fundamental principles for effective token selection: Task Relevance, Information Coverage, and Semantic Diversity. Based on these principles, we propose TOPS, a training-free and model-agnostic pruning module that can be applied to various MLLMs. Extensive experiments on 7 MLLM backbones and 14 benchmarks demonstrate that TOPS outperforms prior methods under diverse pruning settings. Notably, on LLaVA-NeXT, TOPS removes 77.8% of visual tokens while preserving 100.0% and 100.6% performance on its 7B and 13B models, respectively, suggesting that pruning redundant visual tokens can sometimes mitigate hallucination and inspire future lightweight MLLM design.

Problem

• visual token pruning
• multimodal large language models
• computational efficiency
• redundant tokens
• instruction-aware pruning

Innovation

• Token Pruning
• First-Principles
• Multimodal LLMs
• Information-Theoretic Analysis
• Model-Agnostic