VideoSeeker: Incentivizing Instance-level Video Understanding via Native Agentic Tool Invocation

2026-05-15

AI Summary

Current large vision-language models are hindered in instance-level video understanding by the spatiotemporal ambiguity of textual prompts, leading to a disconnect between perception and reasoning. This work proposes VideoSeeker, the first agent-based paradigm driven by visual prompting, which integrates tool invocation and active perception capabilities to enable on-demand retrieval and fine-grained comprehension. The authors develop a fully automated four-stage data synthesis pipeline and employ a combination of cold-start supervised learning and reinforcement learning strategies to train an end-to-end vision-language model on large-scale synthetic data. VideoSeeker achieves an average improvement of 13.7% on instance-level video understanding tasks, significantly outperforming closed-source models such as GPT-4o and Gemini-2.5-Pro, while also demonstrating strong transferability on general video benchmarks.

Abstract

Large Vision-Language Models (LVLMs) have shown significant progress in video understanding, yet they face substantial challenges in tasks requiring precise spatiotemporal localization at the instance level. Existing methods primarily rely on text prompts for human-model interaction, but these prompts struggle to provide precise spatial and temporal references, resulting in poor user experience. Furthermore, current approaches typically decouple visual perception from language reasoning, centering reasoning around language rather than visual content, which limits the model's ability to proactively perceive fine-grained visual evidence. To address these challenges, we propose VideoSeeker, a novel paradigm for instance-level video understanding through visual prompts. VideoSeeker seamlessly integrates agentic reasoning with instance-level video understanding tasks, enabling the model to proactively perceive and retrieve relevant video segments on demand. We construct a four-stage fully automated data synthesis pipeline to efficiently generate large-scale, high-quality instance-level video data. We internalize tool-calling and proactive perception capabilities into the model via cold-start supervision and RL training, building a powerful video understanding model. Experiments demonstrate that our model achieves an average improvement of +13.7% over baselines on instance-level video understanding tasks, surpassing powerful closed-source models such as GPT-4o and Gemini-2.5-Pro, while also showing effective transferability on general video understanding benchmarks. The relevant datasets and code will be released publicly.

Problem

• instance-level video understanding
• spatiotemporal localization
• visual perception
• language reasoning
• human-model interaction

Innovation

• agentic tool invocation
• instance-level video understanding
• visual prompting
• proactive perception
• automated data synthesis