QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding

2026-07-01

AI Summary

This work addresses the challenge of severe temporal redundancy in long video understanding, where dense frame processing is computationally expensive and semantically inefficient. The authors propose a plug-and-play, training-free framework for query- and content-aware keyframe selection that jointly models query relevance and content diversity for the first time. By dynamically allocating a keyframe budget and iteratively selecting frames centered around those most relevant to the query while ensuring semantic richness and diversity, the method can be seamlessly integrated into existing video large language models without fine-tuning. It achieves state-of-the-art performance across multiple long video understanding benchmarks, attaining 67.8% accuracy on LongVideoBench with only 128 frames—surpassing GPT-4o’s 66.7% accuracy using 256 frames.

Abstract

Video understanding is often plagued by severe temporal redundancy, where processing dense frame sequences is both semantically inefficient and computationally expensive. This challenge is further amplified when only a small subset of frames is truly relevant to the given query. In this paper, we propose a Query- and Content-Aware (QCA) keyframe selection framework that can select a compact yet information-rich set of frames from long videos. QCA first partitions the video into temporal segments and estimates the information contribution of each segment by jointly modeling query relevance and content deviation, and dynamically allocates keyframe budget to each segment. Within each segment, QCA anchors on the most query-relevant frame and iteratively incorporates additional frames to maximize diversity while maintaining high semantic relevance to the query. Crucially, our method requires no additional training and can be seamlessly integrated into existing Video-LLMs. Extensive experiments across multiple long video understanding benchmarks demonstrate that our proposed approach achieves state-of-the-art performance and has strong generalization ability. For instance, QCA achieves 67.8\% on LongVideoBench using 128 frames, while GPT-4o achieves 66.7\% using 256 frames. Our codes are available in \href{https://github.com/hktk07/QCA}{GitHub}.

Problem

• keyframe selection
• long video understanding
• temporal redundancy
• query relevance
• video summarization

Innovation

• keyframe selection
• query-aware
• content-aware
• long video understanding
• Video-LLM