VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM
🤖 AI Summary
Current video large language models (Video LLMs) exhibit significant limitations in fine-grained spatiotemporal object understanding, primarily due to the scarcity of high-quality object-level video data and dedicated evaluation benchmarks. To address this, we propose a systematic solution: (1) a novel multi-agent collaborative data engine that constructs VideoInstruct-700K—a large-scale, object-aligned video instruction dataset containing 700,000 samples; (2) VideoRefer, a specialized model integrating spatiotemporal region modeling, object-aware visual encoding, and video–language cross-modal alignment; and (3) VideoRefer-Bench, the first comprehensive benchmark for object-level spatiotemporal referring understanding. Extensive experiments demonstrate that VideoRefer achieves substantial improvements over state-of-the-art methods on video referring tasks and attains new SOTA performance across multiple general video understanding benchmarks, including EgoSchema, NextQA, and WebVid-QA.
Application Category
📝 Abstract
Video Large Language Models (Video LLMs) have recently exhibited remarkable capabilities in general video understanding. However, they mainly focus on holistic comprehension and struggle with capturing fine-grained spatial and temporal details. Besides, the lack of high-quality object-level video instruction data and a comprehensive benchmark further hinders their advancements. To tackle these challenges, we introduce the VideoRefer Suite to empower Video LLM for finer-level spatial-temporal video understanding, i.e., enabling perception and reasoning on any objects throughout the video. Specially, we thoroughly develop VideoRefer Suite across three essential aspects: dataset, model, and benchmark. Firstly, we introduce a multi-agent data engine to meticulously curate a large-scale, high-quality object-level video instruction dataset, termed VideoRefer-700K. Next, we present the VideoRefer model, which equips a versatile spatial-temporal object encoder to capture precise regional and sequential representations. Finally, we meticulously create a VideoRefer-Bench to comprehensively assess the spatial-temporal understanding capability of a Video LLM, evaluating it across various aspects. Extensive experiments and analyses demonstrate that our VideoRefer model not only achieves promising performance on video referring benchmarks but also facilitates general video understanding capabilities.
Problem
Research questions and friction points this paper is trying to address.
Video LLMs
Limited Understanding
Evaluation Methods
Innovation
Methods, ideas, or system contributions that make the work stand out.
Video LLMs
VideoRefer Suite
Temporal-Spatial Detail Recognition
🔎 Similar Papers
No similar papers found.
