ST-SimDiff: Balancing Spatiotemporal Similarity and Difference for Efficient Video Understanding with MLLMs

2026-05-21

AI Summary

Processing long videos with multimodal large language models incurs prohibitive computational costs due to the vast number of visual tokens, and existing compression methods struggle to preserve both critical dynamic changes and spatiotemporal dependencies. To address this, this work proposes ST-SimDiff, a training-free framework that introduces a novel paradigm centered on “redundancy removal via similarity and key event capture via difference.” Specifically, it constructs a spatiotemporal graph to model complex relationships among tokens and simultaneously selects static representative tokens based on similarity while identifying dynamic keyframes through temporal differencing. This approach significantly reduces computational overhead while substantially outperforming state-of-the-art methods in video understanding, establishing a new efficient paradigm that effectively balances static and dynamic content representation.

Abstract

Multimodal Large Language Models (MLLMs) face significant computational overhead when processing long videos due to the massive number of visual tokens required. To improve efficiency, existing methods primarily reduce redundancy by pruning or merging tokens based on importance or similarity. However, these approaches largely overlook a critical dimension of video content, i.e., changes and turning points, and they lack a collaborative model for spatio-temporal relationships. To address this, we propose a new perspective: similarity is for identifying redundancy, while difference is for capturing key events. Based on this, we designed a training-free framework named ST-SimDiff. We first construct a spatio-temporal graph from the visual tokens to uniformly model their complex associations. Subsequently, we employ a parallel dual-selection strategy: 1) similarity-based selection uses community detection to retain representative tokens, compressing static information; 2) temporal difference-based selection precisely locates content-changing points to preserve tokens that capture key dynamic shifts. This allows it to preserve both static and dynamic content with a minimal number of tokens. Extensive experiments show our method significantly outperforms state-of-the-art approaches while substantially reducing computational costs. Our code is available in https://github.com/bingjunluo/ST-SimDiff.

Problem

• video understanding
• computational efficiency
• visual token redundancy
• spatiotemporal modeling
• key event detection

Innovation

• spatiotemporal similarity
• temporal difference
• token selection
• multimodal large language models
• video understanding