KFS-Bench: Comprehensive Evaluation of Key Frame Sampling in Long Video Understanding

2025-12-15

AI Summary

To address the lack of direct, robust evaluation methods for keyframe sampling in long-video question answering (LVQA), this paper introduces KFS-Bench—the first dedicated benchmark for evaluating keyframe sampling strategies in LVQA. Our contributions are threefold: (1) We propose a novel multi-scenario decoupled annotation paradigm, enabling fine-grained ground-truth assessment across diverse video contexts; (2) We define a comprehensive sampling quality metric that jointly accounts for precision, scene coverage, and sampling balance; (3) We develop an adaptive balanced sampling algorithm grounded in cross-modal relevance between questions and video frames. Extensive experiments across multiple multimodal large language models (MLLMs) demonstrate that our method significantly improves both keyframe sampling quality and downstream QA accuracy. Moreover, our proposed metric exhibits strong correlation with model performance. The benchmark, source code, and annotated data are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Question Answering

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

We propose KFS-Bench, the first benchmark for key frame sampling in long video question answering (QA), featuring multi-scene annotations to enable direct and robust evaluation of sampling strategies. Key frame sampling is crucial for efficient long-form video understanding. In long video QA, selecting informative frames enables multimodal large language models (MLLMs) to improve both accuracy and efficiency. KFS-Bench addresses the limitation of prior works that only indirectly assess frame selection quality via QA accuracy. By providing ground-truth annotations of multiple disjoint scenes required per question, KFS-Bench allows us to directly analyze how different sampling approaches capture essential content across an entire long video. Using KFS-Bench, we conduct a comprehensive study of key frame sampling methods and identify that not only sampling precision but also scene coverage and sampling balance are the key factors influencing QA performance. Regarding all the factors, we design a novel sampling quality metric that correlates with QA accuracy. Furthermore, we develop a novel key frame sampling method that leverages question-video relevance to balance sampling diversity against question-frame similarity, thereby improving coverage of relevant scenes. Our adaptively balanced sampling approach achieves superior performance in both key frame sampling and QA performance. The benchmark is available at https://github.com/NEC-VID/KFS-Bench.

Problem

• Evaluates key frame sampling for long video question answering
• Addresses indirect assessment of frame selection quality
• Measures sampling precision, scene coverage, and balance factors

Innovation

• Novel benchmark with multi-scene annotations for direct evaluation
• New sampling quality metric correlating with QA accuracy
• Question-guided adaptive sampling balancing diversity and relevance