AdaRD-key: Adaptive Relevance-Diversity Keyframe Sampling for Long-form Video understanding

2025-10-03

AI Summary

Long video understanding suffers from suboptimal keyframe sampling: uniform sampling often misses event-critical moments, while fixed-interval or purely diversity-driven strategies respectively impair temporal coherence or query relevance. This paper proposes a training-free, adaptive keyframe selection method that jointly optimizes a unified relevance–diversity objective—combining query-conditioned relevance scoring with a log-determinant diversity measure. A lightweight relevance-aware gating mechanism automatically reverts to diversity-only selection under weak query–video alignment. An exclusion window is further introduced to enforce inter-frame spacing, balancing temporal coverage and visual discriminability. The method is plug-and-play, runs in real time on a single GPU, and achieves state-of-the-art performance on LongVideoBench and Video-MME, significantly advancing multimodal large language models’ long-video question-answering capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Video Understanding & Activity Analysis
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Understanding long-form videos remains a significant challenge for vision--language models (VLMs) due to their extensive temporal length and high information density. Most current multimodal large language models (MLLMs) rely on uniform sampling, which often overlooks critical moments, leading to incorrect responses to queries. In parallel, many keyframe selection approaches impose rigid temporal spacing: once a frame is chosen, an exclusion window suppresses adjacent timestamps to reduce redundancy. While effective at limiting overlap, this strategy frequently misses short, fine-grained cues near important events. Other methods instead emphasize visual diversity but neglect query relevance. We propose AdaRD-Key, a training-free keyframe sampling module for query-driven long-form video understanding. AdaRD-Key maximizes a unified Relevance--Diversity Max-Volume (RD-MV) objective, combining a query-conditioned relevance score with a log-determinant diversity component to yield informative yet non-redundant frames. To handle broad queries with weak alignment to the video, AdaRD-Key employs a lightweight relevance-aware gating mechanism; when the relevance distribution indicates weak alignment, the method seamlessly shifts into a diversity-only mode, enhancing coverage without additional supervision. Our pipeline is training-free, computationally efficient (running in real time on a single GPU), and compatible with existing VLMs in a plug-and-play manner. Extensive experiments on LongVideoBench and Video-MME demonstrate state-of-the-art performance, particularly on long-form videos. Code available at https://github.com/Xian867/AdaRD-Key.

Problem

• Addresses inefficient keyframe sampling in long-form video analysis
• Solves rigid temporal spacing missing fine-grained event cues
• Balances query relevance with visual diversity in frame selection

Innovation

• Adaptive keyframe sampling with unified relevance-diversity objective
• Lightweight gating mechanism for query-video alignment
• Training-free real-time plug-and-play video understanding module