Evading Visual Aphasia: Contrastive Adaptive Semantic Token Pruning for Vision-Language Models

2026-05-10

AI Summary

Existing pruning methods for vision-language models rely on shallow attention scores, often erroneously discarding visual tokens critical for compositional reasoning and thereby compromising visual grounding capabilities—leading models into a “visually mute” failure mode dominated by linguistic priors. To address this, we propose a training-free, adaptive semantic routing pruning framework that identifies query-relevant anchor tokens through cross-modal attention, evaluates contextual dispersion via attention entropy, and dynamically balances semantic evidence against complementary spatial context using a contrastive routing score. Departing from conventional single-pass scalar-based token pruning, our approach reduces visual tokens by 77.8% on average across seven benchmarks, achieves a 2.15× speedup in inference latency, and retains 98.64% of the original model performance—significantly outperforming current state-of-the-art methods.

Abstract

Are low-attention visual tokens truly redundant in vision-language reasoning? Existing pruning methods often assume so, ranking visual tokens by shallow text-to-image attention and discarding low-scoring patches to accelerate LVLM inference. We show that this scalar criterion is unreliable for compositional reasoning: tokens ignored in early layers can later become essential for resolving secondary objects, spatial relations, and contextual cues. Premature pruning can therefore induce Visual Aphasia, a failure mode in which the model loses visual grounding and falls back on language priors. We introduce COAST (COntrastive Adaptive Semantic Token Pruning), a training-free pruning framework that casts compression as adaptive semantic routing. COAST uses native cross-modal attention to identify query-specific anchors and estimate contextual dispersion via attention entropy, then adapts the retention trade-off between semantic evidence and spatial context. It further uses a contrastive routing score to preserve both anchor-aligned evidence and complementary spatial context. Across seven benchmarks, COAST reduces visual tokens by 77.8% and achieves a 2.15x latency speedup while retaining 98.64% of the original average performance. Beyond a single backbone or compression setting, COAST consistently outperforms strong pruning baselines across token budgets and generalizes across multiple LVLM families, showing that adaptive semantic routing is a robust alternative to one-shot scalar pruning

Problem

• visual token pruning
• vision-language models
• compositional reasoning
• visual aphasia
• cross-modal attention

Innovation

• token pruning
• vision-language models
• adaptive semantic routing
• contrastive routing
• visual aphasia