Benchmarking Scientific Understanding and Reasoning for Video Generation using VideoScience-Bench

2025-12-02

AI Summary

Current video generation models exhibit significant deficiencies in zero-shot scientific reasoning—particularly in modeling undergraduate-level physics and chemistry principles—while existing benchmarks only cover physical commonsense, failing to rigorously assess scientific understanding. Method: We introduce VideoScience-Bench, the first benchmark designed to evaluate the scientific reasoning capabilities of video generation models. It comprises 200 composite scientific scenarios across 14 physics and chemistry topics and 103 core concepts. We conceptualize video models as “scientific reasoners” and propose a novel five-dimensional evaluation framework, with high-consistency automated scoring enabled by Vision-Language Models-as-a-Judge (VLM-as-a-Judge). Contribution/Results: Under text-to-video (T2V) and image-to-video (I2V) settings, we conduct zero-shot evaluation on seven state-of-the-art models. Our systematic assessment reveals fundamental limitations in modeling dynamic scientific phenomena, establishing a reproducible benchmark and delineating a clear roadmap for developing scientifically aware generative models.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

The next frontier for video generation lies in developing models capable of zero-shot reasoning, where understanding real-world scientific laws is crucial for accurate physical outcome modeling under diverse conditions. However, existing video benchmarks are physical commonsense-based, offering limited insight into video models'scientific reasoning capability. We introduce VideoScience-Bench, a benchmark designed to evaluate undergraduate-level scientific understanding in video models. Each prompt encodes a composite scientific scenario that requires understanding and reasoning across multiple scientific concepts to generate the correct phenomenon. The benchmark comprises 200 carefully curated prompts spanning 14 topics and 103 concepts in physics and chemistry. We conduct expert-annotated evaluations across seven state-of-the-art video models in T2V and I2V settings along five dimensions: Prompt Consistency, Phenomenon Congruency, Correct Dynamism, Immutability, and Spatio-Temporal Continuity. Using a VLM-as-a-Judge to assess video generations, we observe strong correlation with human assessments. To the best of our knowledge, VideoScience-Bench is the first benchmark to evaluate video models not only as generators but also as reasoners, requiring their generations to demonstrate scientific understanding consistent with expected physical and chemical phenomena. Our data and evaluation code are available at: href{https://github.com/hao-ai-lab/VideoScience}{github.com/hao-ai-lab/VideoScience}.

Problem

• Evaluates video models' scientific reasoning and understanding.
• Assesses zero-shot reasoning with real-world scientific laws.
• Measures ability to generate scientifically accurate physical phenomena.

Innovation

• Introduces VideoScience-Bench to evaluate scientific reasoning in video models
• Uses VLM-as-a-Judge for automated assessment of video generations
• Requires models to demonstrate understanding of physics and chemistry concepts