Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models

2025-03-24

AI Summary

This work addresses the lack of dedicated benchmarks for evaluating factual consistency in Large Video-Language Models (LVLMs). We introduce Video SimpleQA, the first benchmark specifically designed for objective factual verification in video understanding, requiring models to integrate external knowledge, perform explicit temporal reasoning, and produce deterministic short answers. We propose a novel five-dimensional evaluation framework—external knowledge reliance, objective factual questioning, deterministic short-answer generation, authoritative source verification, and temporal reasoning—and support automated LLM-as-a-judge evaluation. Leveraging multi-source authoritative annotation, RAG-based comparative analysis, and a cross-model systematic evaluation framework, we empirically assess 41 state-of-the-art LVLMs. Results show that the top-performing model, Gemini-1.5-Pro, achieves only 54.4% F-score; test-time computation yields marginal gains; while RAG consistently improves factual accuracy, it incurs substantial latency overhead.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent advancements in Large Video Language Models (LVLMs) have highlighted their potential for multi-modal understanding, yet evaluating their factual grounding in video contexts remains a critical unsolved challenge. To address this gap, we introduce Video SimpleQA, the first comprehensive benchmark tailored for factuality evaluation of LVLMs. Our work distinguishes from existing video benchmarks through the following key features: 1) Knowledge required: demanding integration of external knowledge beyond the explicit narrative; 2) Fact-seeking question: targeting objective, undisputed events or relationships, avoiding subjective interpretation; 3) Definitive & short-form answer: Answers are crafted as unambiguous and definitively correct in a short format, enabling automated evaluation through LLM-as-a-judge frameworks with minimal scoring variance; 4) External-source verified: All annotations undergo rigorous validation against authoritative external references to ensure the reliability; 5) Temporal reasoning required: The annotated question types encompass both static single-frame understanding and dynamic temporal reasoning, explicitly evaluating LVLMs factuality under the long-context dependencies. We extensively evaluate 41 state-of-the-art LVLMs and summarize key findings as follows: 1) Current LVLMs exhibit notable deficiencies in factual adherence, particularly for open-source models. The best-performing model Gemini-1.5-Pro achieves merely an F-score of 54.4%; 2) Test-time compute paradigms show insignificant performance gains, revealing fundamental constraints for enhancing factuality through post-hoc computation; 3) Retrieval-Augmented Generation demonstrates consistent improvements at the cost of additional inference time overhead, presenting a critical efficiency-performance trade-off.

Problem

• Evaluating factual grounding in Large Video Language Models (LVLMs).
• Creating a benchmark for factuality assessment in video contexts.
• Assessing LVLMs' performance on temporal reasoning and knowledge integration.

Innovation

• Introducing Video SimpleQA benchmark for LVLMs
• External knowledge integration for factuality evaluation
• Automated evaluation using LLM-as-a-judge frameworks