Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models

2026-07-15

AI Summary

This work addresses the high computational cost of vision-language models arising from processing a large number of visual tokens, which hinders deployment on resource-constrained devices. Existing token compression approaches often rely on attention mechanisms or complex similarity computations, limiting their practical acceleration gains. To overcome these limitations, the authors propose a plug-and-play, attention-free, lightweight token compression framework. It introduces information entropy to assess token importance for the first time, thereby eliminating dependence on attention maps. Furthermore, a transformation-induced consistency signal is designed to preserve representational diversity, combined with a stride-based selection strategy for efficient token reduction. The method is compatible with mainstream acceleration frameworks and achieves substantial token reduction across multiple vision-language benchmarks while maintaining an excellent trade-off between accuracy and efficiency.

Abstract

Vision-Language Models (VLMs) have achieved strong performance in multimodal understanding, yet remain challenging to deploy on resource-constrained edge devices due to the substantial computational overhead of processing numerous visual tokens. Token reduction is a promising direction for accelerating VLMs inference, but existing approaches either rely on attention maps that are incompatible with modern acceleration frameworks or depend on computationally intensive pairwise similarity comparisons, which undermine scalability and negate their practical benefits in deployment. In this paper, we propose an attention-free and lightweight token reduction framework as a plug-and-play module for VLMs, which preserves both important and diverse tokens to produce a compact visual representation. First, to enable attention-free importance estimation, we adopt an information-theoretic perspective and quantify token information using a novel entropy-based criterion, retaining those with more expressive and less degenerate feature representations. Second, to ensure diverse visual coverage in a lightweight manner, we introduce a transformation-induced consistency signal where similar tokens yield similar signals, such that sorting by this signal places similar tokens close to each other and enables stride-based selection to produce a diverse token set. Extensive experiments across multiple VLMs benchmarks demonstrate that our framework achieves a favorable accuracy-efficiency trade-off, maintaining competitive performance under aggressive compression.

Problem

• Vision-Language Models
• Token Reduction
• Efficient Inference
• Edge Deployment
• Computational Overhead

Innovation

• attention-free
• token reduction
• entropy-based importance
• diverse token selection
• lightweight VLM