ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs

2026-06-30

AI Summary

This work addresses the high computational cost of multimodal large language models (MLLMs) caused by lengthy visual token sequences and the attention logit collapse commonly induced by existing training-free compression methods. To this end, the authors propose ERA, a novel framework that introduces, for the first time, a logit-preserving mechanism through a three-stage pipeline: Dual-view Entropy-based Pruning (DEP), Bias-aware Token Recovery (BTR), and Logit-Aware Attention Rectification (LAR). Operating without any retraining, ERA efficiently compresses visual tokens while maintaining robust performance across single-image, multi-image, and video tasks. The method significantly reduces inference overhead, is compatible with various mainstream MLLMs, and effectively mitigates attention collapse, thereby offering a practical and scalable solution for efficient multimodal reasoning.

Abstract

Multimodal Large Language Models (MLLMs) incur prohibitive inference costs due to long visual token sequences. Training-free visual token reduction provides an efficient solution. However, existing methods distort attention distributions, giving rise to a phenomenon we term Attention Logit Collapse. To address this issue, we propose ERA, an Entropy-guided visual token pruning framework with Rectified Attention for efficient MLLMs. Specifically, ERA comprises three crucial components: Dual-view Entropy Pruning (DEP), Bias-aware Token Recycling (BTR), and Logit-preserving Attention Rectification (LAR). First, DEP identifies representative anchor tokens by jointly modeling visual diversity and head-wise saliency. BTR then recycles pruned tokens into their corresponding anchors while estimating a cluster-level logit bias. Building upon this, LAR injects the estimated bias into attention logits, effectively rectifying the collapse induced by token reduction. Together, these components preserve visual evidence even under aggressive compression, enabling robust performance across single-image, multi-image, and video settings on a wide range of MLLMs. Beyond delivering practical acceleration, ERA establishes logit-preserving visual token pruning as a principled framework for efficient MLLMs, unifying theoretical foundation, algorithmic design, and practical deployment. The code is at https://github.com/924973292/ERA.

Problem

• Multimodal Large Language Models
• Visual Token Pruning
• Attention Logit Collapse
• Inference Efficiency
• Entropy-guided Pruning

Innovation

• visual token pruning
• attention rectification
• entropy-guided selection
• logit-preserving compression
• multimodal LLMs