QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understanding

2026-07-05

AI Summary

Existing vision-language models struggle with long-form video understanding due to interference from irrelevant segments, while current retrieval methods suffer from biased relevance estimation, insufficient diversity, and temporal collapse. This work proposes QSVideo, a unified framework that introduces a novel Query-Conditioned Semantic Ranker (QSRanker) to structurally model relevance across object, action, and spatial dimensions. By integrating query rewriting, multidimensional semantic modeling, diversity-aware optimization, and a tailored temporal alignment strategy, QSVideo jointly optimizes frame-level informativeness and temporal coverage through its QSRetrieval mechanism. The approach substantially improves the performance of vision-language models under strict frame budget constraints across multiple long-form and streaming video benchmarks.

Abstract

The performance of vision-language models (VLMs) in video understanding declines with increasing video duration, as video moments unrelated to the query confuse their language components. Multimodal retrieval has emerged as a critical component of video understanding, addressing this challenge by localizing key visual evidence. However, existing multimodal retrieval methods suffer from biased relevance estimation, limited diversity, and temporal collapse. In this paper, we propose QSVideo, a unified framework that systematically addresses relevance, diversity, and temporal modeling in video retrieval. We first introduce a query-conditioned semantic ranker, QSRanker, which reformulates arbitrary questions into retrieval-friendly queries and estimates structured relevance along object, action, and location dimensions. Building upon this, we design QSRetrieval to jointly optimize relevance and diversity for more informative frame selection. Moreover, we propose temporal alignment strategies tailored for both long and streaming videos to improve evidence recall. Extensive experiments on long and streaming video benchmarks demonstrate that QSVideo greatly enhances video VLM performance under strict frame limit constraints. The code is available at https://github.com/human-analysis/QSVideo.

Problem

• video understanding
• vision-language models
• multimodal retrieval
• temporal collapse
• relevance estimation

Innovation

• query-conditioned retrieval
• semantic temporal retrieval
• relevance-diversity optimization
• temporal alignment
• video understanding