Video-CoT: A Comprehensive Dataset for Spatiotemporal Understanding of Videos Based on Chain-of-Thought

2025-06-10

AI Summary

Current large-scale vision-language models (VLMs) exhibit weak spatiotemporal fine-grained reasoning on videos and lack interpretable, evaluable benchmarks. Method: We introduce VideoCoT—the first Chain-of-Thought (CoT) benchmark for video spatiotemporal understanding—comprising 192K fine-grained question-answer pairs and 23K human-annotated CoT trajectories. We systematically adapt the CoT paradigm to video understanding via spatiotemporal-aware question generation and a multi-granularity interpretability annotation framework. We further establish a standardized evaluation protocol with 750 images/tasks and comprehensive metrics. Results: Experiments reveal that mainstream VLMs achieve only <42% average accuracy on spatiotemporal reasoning, exposing a fundamental limitation. VideoCoT provides an open-source, reproducible, domain-specific baseline for modeling, evaluating, and advancing interpretable video reasoning research.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Video content comprehension is essential for various applications, ranging from video analysis to interactive systems. Despite advancements in large-scale vision-language models (VLMs), these models often struggle to capture the nuanced, spatiotemporal details essential for thorough video analysis. To address this gap, we introduce Video-CoT, a groundbreaking dataset designed to enhance spatiotemporal understanding using Chain-of-Thought (CoT) methodologies. Video-CoT contains 192,000 fine-grained spa-tiotemporal question-answer pairs and 23,000 high-quality CoT-annotated samples, providing a solid foundation for evaluating spatiotemporal understanding in video comprehension. Additionally, we provide a comprehensive benchmark for assessing these tasks, with each task featuring 750 images and tailored evaluation metrics. Our extensive experiments reveal that current VLMs face significant challenges in achieving satisfactory performance, high-lighting the difficulties of effective spatiotemporal understanding. Overall, the Video-CoT dataset and benchmark open new avenues for research in multimedia understanding and support future innovations in intelligent systems requiring advanced video analysis capabilities. By making these resources publicly available, we aim to encourage further exploration in this critical area. Project website:https://video-cot.github.io/ .

Problem

• Enhancing spatiotemporal understanding in video analysis
• Addressing limitations of vision-language models in video comprehension
• Providing a benchmark for evaluating spatiotemporal reasoning tasks

Innovation

• Introduces Video-CoT dataset for spatiotemporal video understanding
• Uses Chain-of-Thought methodologies for nuanced video analysis
• Provides benchmark with tailored evaluation metrics