TORINO: Token Reduction via Interpretable Concept Overlap in Vision-Language Models

2026-07-05

AI Summary

This work addresses the high computational cost of vision-language models arising from processing a large number of visual tokens and the lack of explicit semantic representation in existing compression methods. The authors propose a plug-and-play, adaptive token compression framework that maps visual tokens into an interpretable latent concept space via sparse autoencoders. For the first time, a concept overlap mechanism is introduced to dynamically group tokens based on shared semantic activations and adaptively prune or merge them without fine-tuning the original model. This approach overcomes the limitations of fixed compression budgets by adapting to varying image complexity, significantly reducing token counts across multiple vision-language benchmarks while achieving an excellent trade-off between efficiency and accuracy with minimal performance degradation.

Abstract

Vision-Language Models (VLMs) have demonstrated impressive capabilities across different tasks, but their computational cost is dominated by the large number of visual tokens fed to the language model. Existing token reduction methods rely on attention-based scores or pairwise similarity, without an explicit semantic representation of each token. We introduce TORINO (TOken Reduction via Interpretable coNcept Overlap), a plug-and-play framework for adaptive visual token reduction in VLMs that requires no fine-tuning of the underlying model. TORINO leverages Sparse Autoencoders (SAEs) to project visual tokens into an interpretable latent space where token relationships can be analyzed through shared concept activations. Specifically, we define concept overlap as the degree of agreement between active SAE latents and use it to group tokens that share semantic content. Reduction within each group is then performed by either pruning or merging, providing a unified framework that preserves semantically important visual information while removing redundancy. Unlike fixed-budget approaches, TORINO dynamically adapts the reduction rate to input complexity, allowing different images to retain different numbers of tokens. Experiments across multiple vision-language benchmarks show that TORINO achieves favorable efficiency-accuracy trade-offs, reducing the number of visual tokens with minimal performance loss.

Problem

• Vision-Language Models
• Token Reduction
• Computational Cost
• Semantic Redundancy
• Visual Tokens

Innovation

• Token Reduction
• Interpretable Latent Space
• Concept Overlap
• Sparse Autoencoders
• Vision-Language Models