MME-CoF-Pro: Evaluating Reasoning Coherence in Video Generative Models with Text and Visual Hints

2026-03-20

AI Summary

Current video generation models lack effective evaluation of cross-frame causal consistency, making it difficult to assess their reasoning coherence. To address this gap, this work proposes the first evaluation framework specifically designed for reasoning coherence in video generation, introducing MME-CoF-Pro—a benchmark comprising 303 samples across 16 task categories under three settings: no prompt, text prompt, and visual prompt. The framework also introduces a process-level Reasoning Score metric. Through controlled three-stage prompting experiments and multi-model comparative evaluations, the study reveals that existing models generally exhibit weak reasoning coherence, which is often decoupled from generation quality: text prompts frequently induce hallucinatory reasoning, while visual prompts, though beneficial for structured perception, show limited effectiveness on fine-grained tasks.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Visual Reasoning & Symbolic Representations

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Video generative models show emerging reasoning behaviors. It is essential to ensure that generated events remain causally consistent across frames for reliable deployment, a property we define as reasoning coherence. To bridge the gap in literature for missing reasoning coherence evaluation, we propose MME-CoF-Pro, a comprehensive video reasoning benchmark to assess reasoning coherence in video models. Specifically, MME-CoF-Pro contains 303 samples across 16 categories, ranging from visual logical to scientific reasoning. It introduces Reasoning Score as evaluation metric for assessing process-level necessary intermediate reasoning steps, and includes three evaluation settings, (a) no hint (b) text hint and (c) visual hint, enabling a controlled investigation into the underlying mechanisms of reasoning hint guidance. Evaluation results in 7 open and closed-source video models reveals insights including: (1) Video generative models exhibit weak reasoning coherence, decoupled from generation quality. (2) Text hints boost apparent correctness but often cause inconsistency and hallucinated reasoning (3) Visual hints benefit structured perceptual tasks but struggle with fine-grained perception. Website: https://video-reasoning-coherence.github.io/

Problem

• reasoning coherence
• video generative models
• causal consistency
• reasoning evaluation
• video reasoning

Innovation

• reasoning coherence
• video generative models
• Reasoning Score
• visual hint
• text hint