Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models

2026-06-23

AI Summary

This work addresses the inefficiency in multimodal large language models caused by redundant visual tokens during inference and proposes a training-free token pruning framework. The method introduces, for the first time, Cross-Layer Spectral Evolution (CLSE) as a metric for token importance, which tracks the dynamic changes of visual tokens’ frequency-domain representations across Transformer layers to identify and retain semantically active tokens. This approach overcomes the positional bias inherent in single-layer signal analysis. Experimental results demonstrate that the proposed strategy substantially reduces FLOPs, key-value cache memory usage, and inference latency on both image and video benchmarks, while preserving or even improving model accuracy.

Abstract

Reducing visual token redundancy is critical for accelerating Multimodal Large Language Models (MLLMs) without degrading cross-modal reasoning performance. Existing token pruning methods typically rely on single-layer signals, such as attention scores or token similarities, which overlook the cross-layer transformation of visual representations and may exhibit positional bias in multimodal token sequences. To address this limitation, we propose a training-free token pruning framework based on Cross-Layer Spectral Evolution (CLSE). Instead of measuring token importance from single-layer feature magnitudes, CLSE quantifies how token representations evolve across Transformer layers in the frequency domain. This evolution reflects the transition from high-frequency structural details to low-frequency semantic abstractions. We observe that tokens with stronger spectral redistribution across layers are more likely to be semantically active and should therefore be preserved. By modeling cross-layer token dynamics, CLSE provides a stable importance criterion that mitigates positional bias. Extensive experiments on both image and video benchmarks demonstrate that CLSE achieves a superior trade-off between efficiency and accuracy under aggressive token reduction. Across multiple MLLMs, CLSE reduces FLOPs, KV cache memory, and latency while maintaining competitive or improved performance.

Problem

• token pruning
• multimodal large language models
• visual token redundancy
• cross-layer representation
• positional bias

Innovation

• token pruning
• spectral evolution
• multimodal large language models
• cross-layer dynamics
• training-free