Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs

2026-04-22

AI Summary

Existing training-free visual token pruning methods suffer significant performance degradation in fine-grained video understanding tasks—such as hallucination assessment—due to their inability to preserve critical semantic evidence. This work identifies, for the first time, the presence of “sink tokens”: redundant tokens devoid of semantic content yet disproportionately attracting attention, which critically undermines pruning efficacy. To address this, we propose Sink-Token-aware Pruning (SToP), a plug-and-play method that leverages a sink score to detect and suppress such tokens without requiring any retraining. SToP seamlessly integrates into diverse spatiotemporal pruning frameworks (e.g., VisionZip, FastVid) and substantially enhances fine-grained comprehension across multiple challenging tasks—including hallucination evaluation, open-ended generation, compositional reasoning, and multiple-choice question answering—maintaining high accuracy even when retaining only 10% of the original visual tokens.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Video Large Language Models (Video LLMs) incur high inference latency due to a large number of visual tokens provided to LLMs. To address this, training-free visual token pruning has emerged as a solution to reduce computational costs; however, existing methods are primarily validated on Multiple-Choice Question Answering (MCQA) benchmarks, where coarse-grained cues often suffice. In this work, we reveal that these methods suffer a sharp performance collapse on fine-grained understanding tasks requiring precise visual grounding, such as hallucination evaluation. To explore this gap, we conduct a systematic analysis and identify sink tokens--semantically uninformative tokens that attract excessive attention--as a key obstacle to fine-grained video understanding. When these sink tokens survive pruning, they distort the model's visual evidence and hinder fine-grained understanding. Motivated by these insights, we propose Sink-Token-aware Pruning (SToP), a simple yet effective plug-and-play method that introduces a sink score to quantify each token's tendency to behave as a sink and applies this score to existing spatial and temporal pruning methods to suppress them, thereby enhancing video understanding. To validate the effectiveness of SToP, we apply it to state-of-the-art pruning methods (VisionZip, FastVid, and Holitom) and evaluate it across diverse benchmarks covering hallucination, open-ended generation, compositional reasoning, and MCQA. Our results demonstrate that SToP significantly boosts performance, even when pruning up to 90% of visual tokens.

Problem

• fine-grained video understanding
• visual token pruning
• sink tokens
• video LLMs
• visual grounding

Innovation

• sink tokens
• token pruning
• fine-grained video understanding
• video LLMs
• attention distortion