Moving Beyond Diversity: Visual Token Pruning as Subspace Reconstruction for Efficient VLMs

2026-06-17

AI Summary

This work addresses the high computational cost of current vision-language models (VLMs) caused by excessive visual tokens, a challenge exacerbated by existing diversity-maximization pruning methods that disregard feature magnitudes and thus struggle to accurately reconstruct original representations—particularly in compositional, multi-skill reasoning tasks. The authors reformulate visual token pruning as a column subset selection problem and introduce a training-free, reconstruction-driven pruning paradigm. By iteratively selecting tokens with large projection residuals to minimize subspace reconstruction error and incorporating an “anti-correlation” criterion for context-aware token filtering, the method transcends conventional angular diversity constraints. It achieves state-of-the-art performance across multiple VLMs and benchmarks; notably, when applied to LLaVA, it removes 94% of visual tokens while retaining 95% of original performance, with especially pronounced gains on compositional tasks.

Abstract

Despite their remarkable performance, Vision Language Models (VLMs) incur substantial computational overhead due to the large number of visual tokens. While diversity maximization has become a dominant strategy for token reduction, existing methods rely on cosine-based normalized similarity that discards magnitude information, failing to faithfully approximate the original feature representation and leading to suboptimal performance, particularly on compositional multi-skill reasoning tasks. In this paper, we introduce SPARE, a subspace reconstruction method that reformulates token pruning as a column subset selection problem and explicitly minimizes reconstruction error. By iteratively selecting tokens with large projection residuals, SPARE performs reconstruction-driven pruning beyond angular diversity. Moreover, we reveal a counterintuitive anti-relevance phenomenon: tokens with lower image-text relevance score can better preserve contextual information. Based on this finding, we incorporate anti-relevance into SPARE as an additional selection criterion to promote context-aware token selection. Extensive experiments across multiple VLMs and benchmarks demonstrate that SPARE consistently achieves state-of-the-art performance, with strong gains on compositional tasks. When applied to LLaVA, SPARE removes up to 94% of visual tokens while retaining 95% of the baseline performance, all in a fully training-free manner.

Problem

• Vision Language Models
• Token Pruning
• Feature Reconstruction
• Diversity Maximization
• Compositional Reasoning

Innovation

• visual token pruning
• subspace reconstruction
• column subset selection
• anti-relevance
• training-free