Fre-Res: Frequency-Residual Video Token Compression for Efficient Video MLLMs

2026-05-10

arXiv.org

AI Summary

This work addresses the inherent tension in video multimodal large language models between high spatial fidelity and broad temporal coverage: fine-grained visual details demand numerous spatial tokens, while capturing transient events requires dense temporal sampling. To resolve this trade-off, the authors propose Fre-Res, a dual-path framework that decouples spatiotemporal information by preserving sparse, high-fidelity spatial anchor tokens and compactly modeling temporal dynamics via residual frequency tokens. Innovatively, a 1D discrete cosine transform (DCT) is applied to temporal residual trajectories in the visual latent space, leveraging their energy concentration in low-frequency components for efficient token compression. A spatially guided absorber then precisely injects these compressed temporal residuals into their corresponding spatial anchors. Coupled with adaptive token budget allocation, Fre-Res substantially reduces visual token counts on both short- and long-video benchmarks while matching or approaching the performance of full-token baselines, demonstrating that frequency-domain residuals effectively preserve causal transition cues and spatial anchors enable fine-grained reasoning.

Abstract

Video MLLMs face a persistent tension between spatial fidelity and temporal coverage: preserving fine-grained visual details requires many spatial tokens, while capturing short-lived events requires dense temporal sampling. We propose \textbf{Fre-Res}, a budget-adaptive dual-track video-token compression framework that separates these two forms of evidence. Fre-Res preserves sparse high-fidelity spatial anchors and represents dense temporal evolution through compact residual-frequency tokens. Specifically, it applies temporal 1D-DCT to inter-frame residual trajectories in vision-latent space, where we observe strong low-frequency concentration. To align frequency-domain dynamics with native visual embeddings, Fre-Res introduces a Spatial-Guided Absorber that injects temporal residual information into spatially corresponding anchor tokens. Across fine-grained short-video and long-video reasoning benchmarks, Fre-Res achieves a favorable accuracy--efficiency trade-off, matching or approaching full-token performance while substantially reducing visual-token length. Extensive ablations further show that temporal-frequency residuals preserve causal transition cues, while spatial anchors remain essential for fine-grained object and layout reasoning.

Problem

• Video MLLMs
• spatial fidelity
• temporal coverage
• token compression
• visual details

Innovation

• video token compression
• frequency-residual representation
• spatial-temporal disentanglement
• 1D-DCT
• budget-adaptive MLLM