VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement

2026-07-01

AI Summary

This work addresses a critical limitation in existing video retrieval approaches, which treat retrieval as a one-off preprocessing step and cannot recover from initial failures, thereby undermining downstream fine-grained tasks such as temporal localization. Conversely, current agent-based frameworks often assume relevant videos are already provided, overlooking the challenges of open-domain retrieval. To bridge this gap, the paper proposes VideoSearch-R1, an iterative retrieval-reasoning agent framework that enables multi-turn interaction with a video search engine. It introduces a novel Soft Query Refinement (SQR) mechanism that dynamically optimizes query representations in a continuous latent space, coupled with Group Relative Policy Optimization (GRPO) guided by task-specific rewards to jointly train retrieval and reasoning components. Evaluated on three VCMR benchmarks, VideoSearch-R1 achieves state-of-the-art performance, with SQR yielding more efficient and accurate query refinement than explicit textual rewriting while using fewer tokens.

Abstract

As video corpora continue to expand in both scale and task complexity, there is increasing demand for approaches that retrieve relevant videos from large-scale corpora (inter-video reasoning) and subsequently perform fine-grained, query-conditioned tasks (intra-video reasoning) within the retrieved content, such as temporal grounding. However, existing approaches typically treat retrieval as a preprocessing step, and consequently, when the initial retrieval fails, there is no mechanism to refine the search, leading to the failure of subsequent fine-grained intra-video reasoning. Moreover, while recent agentic frameworks have advanced video understanding, they typically assume that the query-relevant video is already given, focusing exclusively on intra-video reasoning tasks. To address these limitations, we propose VideoSearch-R1, an agentic framework for iterative video retrieval and reasoning through multi-turn interaction with a video search engine. Specifically, we introduce Soft Query Refinement (SQR) to refine search query tokens in a continuous latent space rather than rewriting queries in the discrete text space, enabling more efficient and fine-grained adjustments. SQR and its reasoning process are trained using Group Relative Policy Optimization (GRPO), guided by task-level reward signals derived from retrieval and downstream tasks. Building upon this, VideoSearch-R1 achieves state-of-the-art performance across three datasets on Video Corpus Moment Retrieval (VCMR), iteratively retrieving videos from large-scale corpora, refining search queries, and performing precise query-conditioned temporal grounding within the retrieved content. Our analyses show that SQR effectively refines the original query, requiring significantly fewer generated tokens than explicit text-level query refinement. Code and model checkpoints are publicly available at mlvlab.github.io/VideoSearch-R1.

Problem

• video retrieval
• temporal grounding
• query refinement
• iterative reasoning
• large-scale video corpora

Innovation

• Soft Query Refinement
• Iterative Video Retrieval
• Video Corpus Moment Retrieval
• Agentic Framework
• Group Relative Policy Optimization