Video Active Perception: Effective Inference-Time Long-Form Video Understanding with Vision-Language Models

2026-05-02

AI Summary

This work addresses the inefficiency and performance limitations of large vision-language models in long-form video question answering due to suboptimal frame sampling. To this end, we propose VAP, the first training-free active perception framework. Inspired by active perception theory, VAP formulates keyframe selection as a data acquisition process, leveraging a lightweight text-conditioned video generation model to encode prior knowledge and dynamically select the most question-relevant frames during inference. This approach substantially improves both frame utilization efficiency and reasoning capability, achieving zero-shot state-of-the-art performance on multiple long-video QA benchmarks—including EgoSchema and NExT-QA—with up to a 5.6× improvement in frame efficiency compared to models such as GPT-4o.

Abstract

Large vision-language models (VLMs) have advanced multimodal tasks such as video question answering (QA). However, VLMs face the challenge of selecting frames effectively and efficiently, as standard uniform sampling is expensive and performance may plateau. Inspired by active perception theory, which posits that models gain information by acquiring data that differs from their expectations, we introduce Video Active Perception (VAP), a training-free method to enhance long-form video QA using VLMs. Our approach treats keyframe selection as data acquisition in active perception and leverages a lightweight text-conditioned video generation model to represent prior world knowledge. Empirically, VAP achieves state-of-the-art zero-shot results on long-form or reasoning video QA datasets such as EgoSchema, NExT-QA, ActivityNet-QA, IntentQA, and CLEVRER, achieving an increase of up to 5.6 x frame efficiency by frames per question over standard GPT-4o, Gemini 1.5 Pro, and LLaVA-OV. Moreover, VAP shows stronger reasoning abilities than previous methods and effectively selects keyframes relevant to questions. These findings highlight the potential of leveraging active perception to improve the frame effectiveness and efficiency of long-form video QA.

Problem

• video question answering
• keyframe selection
• long-form video understanding
• vision-language models
• active perception

Innovation

• Video Active Perception
• active perception
• vision-language models
• keyframe selection
• long-form video understanding