MMVU: Measuring Expert-Level Multi-Discipline Video Understanding

2025-01-21

AI Summary

Existing video understanding benchmarks focus primarily on basic visual perception and lack rigorous evaluation of domain-specific expertise and higher-order reasoning. Method: This paper introduces MMVU—the first interdisciplinary, expert-level video evaluation benchmark—spanning 27 disciplines across science, medicine, humanities & social sciences, and engineering, comprising 3,000 expert-authored questions. It pioneers a novel evaluation paradigm integrating zero-shot annotation, explicit reasoning-path embedding, and domain-knowledge augmentation, enabled by multi-disciplinary expert collaboration, stringent quality control, and structured rationale injection. Contribution/Results: Systematic evaluation across 32 state-of-the-art multimodal large language models reveals that even the strongest System-2 models (e.g., o1, Gemini 2.0 Flash Thinking) significantly underperform human experts on knowledge-intensive video understanding tasks—highlighting critical bottlenecks in domain-specific reasoning and deep semantic comprehension.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

We introduce MMVU, a comprehensive expert-level, multi-discipline benchmark for evaluating foundation models in video understanding. MMVU includes 3,000 expert-annotated questions spanning 27 subjects across four core disciplines: Science, Healthcare, Humanities&Social Sciences, and Engineering. Compared to prior benchmarks, MMVU features three key advancements. First, it challenges models to apply domain-specific knowledge and perform expert-level reasoning to analyze specialized-domain videos, moving beyond the basic visual perception typically assessed in current video benchmarks. Second, each example is annotated by human experts from scratch. We implement strict data quality controls to ensure the high quality of the dataset. Finally, each example is enriched with expert-annotated reasoning rationals and relevant domain knowledge, facilitating in-depth analysis. We conduct an extensive evaluation of 32 frontier multimodal foundation models on MMVU. The latest System-2-capable models, o1 and Gemini 2.0 Flash Thinking, achieve the highest performance among the tested models. However, they still fall short of matching human expertise. Through in-depth error analyses and case studies, we offer actionable insights for future advancements in expert-level, knowledge-intensive video understanding for specialized domains.

Problem

• Interdisciplinary Video Understanding
• Expertise Content Analysis
• Model Performance Enhancement

Innovation

• MMVU
• Expert-level Interdisciplinary Video Understanding
• Model Evaluation and Improvement