Incentivizing Vision Language Models to Search for Long Video Question Answering

2026-07-03

AI Summary

This work addresses the limitations of existing models in long-form video question answering, which often process inputs passively and struggle to actively retrieve relevant visual context. To overcome this, the authors propose VSeek, a framework that formulates question answering as an iterative, natural language–driven process of retrieval and reasoning. VSeek introduces a novel neuro-symbolic mechanism that compiles natural language questions into temporal logic specifications, which in turn generate verifiable atomic visual primitives. These primitives provide dense and verifiable reward signals for reinforcement learning, enabling joint optimization of query generation and answer reasoning. Evaluated on standard long-video understanding benchmarks, VSeek significantly outperforms current state-of-the-art methods, achieving an 8% absolute improvement in Pass@1 accuracy and a 15% gain in Pass@4.

Abstract

We introduce VSeek, an agentic framework that transforms long-video question answering (LVQA) from a passive, single-pass perception task into a multi-turn retrieval process. VSeek utilizes a natural language-driven search to identify relevant context within long videos and is post-trained with reinforcement learning (RL) to jointly formulate targeted search queries and reason over retrieved clips for LVQA. While RL post-training has revolutionized reasoning in symbolic domains such as mathematics and code, its application to long-video understanding remains hindered by a lack of verified rewards. To ensure that the retrieved context is relevant, we propose a novel neuro-symbolic approach that bridges open-ended natural language with discrete visual verification. Specifically, complex user queries are compiled into formal temporal logic specifications for systematically decomposing natural language questions into a definitive checklist of required atomic visual primitives, such as key objects and activities, along with their temporal ordering. These systematically derived grounding events provide the critical feedback signal for RL post-training, enabling dense, verifiable rewards based on the successful retrieval of these specific visual elements rather than relying entirely on outcome-only answer accuracy. By explicitly optimizing for this verifiable evidence-seeking behavior, VSeek improves Pass@1 scores by up to 8% and Pass@4 scores by 15% on long-video understanding benchmarks compared to base models. We open-source our code at https://utaustin-swarmlab.github.io/VSeek.

Problem

• long-video question answering
• vision language models
• retrieval
• reinforcement learning
• verifiable rewards

Innovation

• Vision-Language Models
• Long Video Question Answering
• Reinforcement Learning
• Neuro-Symbolic Reasoning
• Temporal Logic