LVBench: An Extreme Long Video Understanding Benchmark

2024-06-12

arXiv.org

AI Summary

Existing short-video understanding benchmarks inadequately support real-world applications requiring hour-scale video comprehension—such as embodied intelligence, in-depth film criticism, and live sports commentary. To address this gap, we introduce LVBench, the first benchmark dedicated to *extreme-long-video understanding* (i.e., videos spanning several hours). LVBench systematically defines an evaluation paradigm encompassing diverse publicly available multi-source videos and multiple tasks—including temporal reasoning, event provenance tracing, and fine-grained question answering—with explicit emphasis on modeling cross-hour semantic coherence. Leveraging structured annotations and a hierarchical evaluation protocol (span-level, global-level, and temporal-dependency-level), it enables end-to-end assessment of multimodal large models. Empirical results show that state-of-the-art models underperform human annotators by 32.7% on average, confirming the benchmark’s substantial difficulty. The full codebase and dataset are publicly released.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent progress in multimodal large language models has markedly enhanced the understanding of short videos (typically under one minute), and several evaluation datasets have emerged accordingly. However, these advancements fall short of meeting the demands of real-world applications such as embodied intelligence for long-term decision-making, in-depth movie reviews and discussions, and live sports commentary, all of which require comprehension of long videos spanning several hours. To address this gap, we introduce LVBench, a benchmark specifically designed for long video understanding. Our dataset comprises publicly sourced videos and encompasses a diverse set of tasks aimed at long video comprehension and information extraction. LVBench is designed to challenge multimodal models to demonstrate long-term memory and extended comprehension capabilities. Our extensive evaluations reveal that current multimodal models still underperform on these demanding long video understanding tasks. Through LVBench, we aim to spur the development of more advanced models capable of tackling the complexities of long video comprehension. Our data and code are publicly available at: https://lvbench.github.io.

Problem

• Addressing long video understanding beyond short clips
• Evaluating multimodal models on extended comprehension tasks
• Challenging models with hours-long video memory demands

Innovation

• Introduces LVBench benchmark for long videos
• Comprises diverse tasks for extended comprehension
• Challenges models with long-term memory capabilities