One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding

2026-04-15

AI Summary

Long video understanding is hindered by the limited context length of vision-language models, which struggle to efficiently process dense temporal information. This work proposes an extreme compression framework that integrates learnable progressive token compression (LP-Comp) with a question-conditioned frame selection mechanism (QC-Comp), substantially reducing computational overhead while preserving critical semantics. The approach combines local attention, segment-wise processing, supervised fine-tuning, and an internal attention-score-based frame filtering strategy, effectively mitigating positional bias in large language models when handling long contexts. Remarkably, with only 2.5% of the original fine-tuning data, the method improves accuracy on LVBench from 42.9% to 46.2% and achieves significant performance gains across multiple long-video benchmarks.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Long video understanding is inherently challenging for vision-language models (VLMs) because of the extensive number of frames. With each video frame typically expanding into tens or hundreds of tokens, the limited context length of large language models (LLMs) forces the VLMs to perceive the frames sparsely and lose temporal information. To address this, we explore extreme video token compression towards \emph{one token per frame} at the final LLM layer. Our key insight is that heuristic-based compression, widely adopted by previous methods, is prone to information loss, and this necessitates supervising LLM layers into \emph{learnable} and \emph{progressive} modules for \emph{token-level compression} (LP-Comp). Such compression enables our VLM to digest 2x-4x more frames with improved performance. To further increase the token efficiency, we investigate \emph{frame-level compression}, which selects the frames most relevant to the queries via the internal attention scores of the LLM layers, named \emph{question-conditioned compression} (QC-Comp). As a notable distinction from previous studies, we mitigate the position bias of LLM attention in long contexts, \emph{i.e.}, the over-concentration on the beginning and end of a sequence, by splitting long videos into short segments and employing local attention. Collectively, our combined \emph{token-level} and \emph{frame-level} leads to an e\textbf{x}treme compression model for long video understanding, named \textbf{\name}, achieving a significantly larger compression ratio and enabling denser frame sampling. Our \name is finetuned from VideoChat-Flash with a data-efficient \emph{supervised compression tuning} stage that only requires 2.5\% of the supervised fine-tuning data, yet boosts the accuracy from 42.9\% to 46.2\% on LVBench and enhances multiple other long video benchmarks.

Problem

• long video understanding
• vision-language models
• token compression
• context length limitation
• temporal information loss

Innovation

• token-level compression
• frame-level compression
• question-conditioned compression
• local attention
• supervised compression tuning