VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding

2026-05-21

AI Summary

Existing video understanding benchmarks struggle to evaluate models’ capabilities in continuous reasoning, information integration, and memory retention over hour-long temporal contexts. To address this gap, this work proposes VideoOdyssey—the first comprehensive diagnostic benchmark supporting multi-granular, ultra-long-context, and fully multimodal (visual-only V and synchronized audio-visual AV) evaluation. It comprises authentic long-form videos spanning 11 domains with an average duration of 109 minutes and introduces “continuous evidence span” as a novel core metric for cognitive load. A structured evaluation framework is established through multi-level question annotations, audio-visual alignment verification, and fine-grained difficulty stratification. Empirical results reveal significant limitations in current multimodal large language models regarding ultra-long-context retrieval, cross-scale reasoning, fine-grained perception, and non-linguistic multimodal comprehension, thereby charting critical directions for future research.

Abstract

Real-world long video understanding requires models to perform continuous tracking, information integration and memory retention over massive temporal spans within extreme video durations. Mastering this intense cognitive load constitutes the fundamental bottleneck in long video understanding. While existing benchmarks have driven progress by scaling up video duration, their evaluation tasks often require comprehending only short and isolated video segments, falling short of capturing the challenge of ultra-long-context reasoning. To measure this cognitive load, we emphasize continuous certificate length, defined as the video length a human must continuously watch to definitively answer a given question. Driven by this metric, we introduce VideoOdyssey, a benchmark specifically designed for ultra-long-context and omni-modal video understanding. VideoOdyssey is characterized by three key features: 1) Extreme video duration and diversity: spanning 11 domains and 54 subcategories with an average video duration of 109 minutes; 2) Comprehensive evaluation scenarios: offering two subsets to address different research focuses, i.e., VideoOdyssey-V for probing the limits of visual understanding in MLLMs, and VideoOdyssey-AV for evaluating synchronized audio-visual understanding for omni-modal models; 3) Ultra-long and multi-level continuous certificates: extending the average continuous certificate to 16 minutes for VideoOdyssey-V and 12.8 minutes for VideoOdyssey-AV. Crucially, we design 5 granular levels from seconds to hours, providing a comprehensive diagnostic tool to evaluate models across varying context lengths and cognitive loads. Extensive evaluations show that bottlenecks of current MLLMs extend beyond simple retrieval to include struggles with continuous reasoning across varying context lengths, fine-grained perception, and non-verbal omni-modal understanding.

Problem

• ultra-long-context
• video understanding
• continuous reasoning
• omni-modal
• cognitive load

Innovation

• ultra-long-context
• omni-modal
• continuous certificate length
• video understanding benchmark
• multi-level evaluation