Spectral Heat Flow for Conservative Token Condensation in Vision-Language Models

2026-07-12

AI Summary

This work addresses the high computational cost of long visual token sequences in vision-language model inference, where existing pruning methods often degrade performance under high compression ratios by disrupting spatial structure and reducing token diversity. To overcome this, the authors propose SpecFlow, a training-free, conservative token compression framework that models visual tokens as nodes in a k-nearest neighbor graph and computes an importance field via spectral heat flow. It adaptively partitions the spatial domain to allocate compression budgets and aggregates discarded information into coreset sink nodes, preserving structural coherence and statistical conservation. Breaking from conventional destructive pruning paradigms, SpecFlow maintains stable performance at high compression rates without fine-tuning. Experiments demonstrate consistent superiority across diverse tasks, models, and compression ratios—for instance, LLaVA-1.5 retains 95.6% of its original performance after removing 88.9% of visual tokens.

Abstract

Vision-Language Models (VLMs) are costly at inference time because they must process long sequences of visual tokens. Existing token pruning methods often degrade under high compression by blindly discarding information, breaking spatial structure or collapsing diversity. We propose SpecFlow, a training-free framework that shifts the paradigm from destructive pruning to conservative condensation, strictly enforcing spatial coverage and statistical conservation to ensure stability. Treating visual tokens as nodes in a $k$NN graph, SpecFlow (i) computes a stable importance field via spectral heat flow to preserve structural coherence, (ii) allocates budgets via adaptive spatial partitioning to guarantee coverage, and (iii) aggregates discarded information into coreset sinks to maintain statistical conservation. The method is plug-and-play, requires no fine-tuning, and is compatible with FlashAttention. Experiments confirm that our SpecFlow outperforms SOTA methods across tasks, VLM architectures, and pruning ratios. Notably, LLaVA-1.5 with SpecFlow retains 95.6\% of original performance despite pruning 88.9\% of visual tokens, offering an exceptional efficiency-accuracy balance. Code is available at https://github.com/Lzy-dot/SpecFlow

Problem

• Vision-Language Models
• token condensation
• inference efficiency
• spatial structure
• statistical conservation

Innovation

• spectral heat flow
• token condensation
• vision-language models
• spatial coverage
• statistical conservation