Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction

2026-06-28

AI Summary

Current video understanding benchmarks predominantly emphasize shallow perceptual tasks, failing to adequately assess a model’s ability to acquire deep procedural knowledge from instructional videos and generalize it to long-horizon GUI agent tasks. To address this gap, this work introduces VG-GUIBench, the first unified evaluation framework that bridges video question answering (VideoQA) and video-guided agent tasks. Central to this framework is TASKER, a task-driven, scene-aware keyframe extraction algorithm that jointly models task relevance and scene dynamics to select informative frames. Experimental results demonstrate that TASKER outperforms the best baseline by 2.0% on EgoSchema and by 1.8% on NExT-QA, significantly enhancing performance in both video question answering and downstream agent tasks.

Abstract

Video understanding is a fundamental capability for multimodal intelligence, and recent Multimodal Large Language Models (MLLMs) have achieved remarkable performance on Video Question Answering (VideoQA) benchmarks. However, existing benchmarks primarily evaluate whether models can perceive shallow visual cues, while rarely examining whether MLLMs can learn deeper knowledge or procedural skills from video tutorials and generalize them to downstream long-horizon agentic tasks. To address this gap, we introduce VG-GUIBench (Video-Guided GUI Benchmark), a new benchmark designed to evaluate whether MLLM-based GUI agents can follow video tutorials to complete corresponding GUI interactive tasks. Furthermore, we observe that the performance of models on both VideoQA and video-guided agentic tasks critically depends on effective keyframe extraction. Based on this observation, we propose TASKER (Task-driven And Scene-aware Keyframe searchER), a keyframe extraction algorithm that jointly considers task relevance and scene dynamics to identify informative frames. Experimental results demonstrate that TASKER achieves significant performance improvements on both VideoQA and video-guided agentic task benchmarks, outperforming the best baseline by 2.0% on the EgoSchema fullset and 1.8% on the NExT-QA dataset, respectively. These results further highlight the potential of generalized keyframe extraction methods for video understanding tasks. Our code and data are available at https://github.com/VG-GUI-TASKER/VG-GUI-TASKER.

Problem

• Video Understanding
• Multimodal Large Language Models
• Video Question Answering
• Agentic Tasks
• Generalization

Innovation

• keyframe extraction
• video understanding
• multimodal LLMs
• agentic tasks
• task-driven