Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning

2026-06-26

AI Summary

Existing video benchmarks struggle to accurately evaluate the temporal logical reasoning capabilities of multimodal large language models in dynamic visual settings, often conflating such reasoning with scene complexity or static recognition. This work proposes Video-MME-Logical, the first systematically designed and controllable diagnostic benchmark that disentangles temporal logical reasoning into five core operations: state tracking, sequential counting, temporal ordering, dynamic spatial reasoning, and structural composition. These are further refined into 25 tunable-difficulty tasks. The benchmark introduces a controllable object-state generation mechanism, intermediate reasoning trajectory validation, and a large-scale synthetic data fine-tuning strategy. Experiments reveal that even state-of-the-art models significantly underperform humans on this benchmark, with a notable gap persisting despite fine-tuning on 500,000 synthetic samples, underscoring its value for diagnosing and guiding model improvement.

Abstract

Recent interest in multimodal large language models (MLLMs) raises a central question: can they reason over dynamic visual evidence rather than merely recognize objects or events in individual frames? This ability, which we refer to as video temporal-logical reasoning, requires models to maintain, update, and compose evidence as visual states evolve across frames. Existing video benchmarks often conflate this capability with scene complexity, static recognition, or uncontrolled temporal variation. To isolate this capability, we introduce Video-MME-Logical, a controlled benchmark organized around five temporal-logical operations: state tracking, sequential counting, temporal ordering, dynamic spatiality, and structural composition. The benchmark contains 25 fine-grained task categories generated with controlled object states, transitions, temporal dependencies, and logical compositions. It enables difficulty-controlled final-answer evaluation by varying temporal horizon and reasoning complexity, and supports intermediate-state diagnostics by verifying whether models recover the required logical reasoning trace before producing the final answer. Experiments with state-of-the-art MLLMs reveal a substantial human-model gap, especially as temporal-logical complexity increases. Supervised fine-tuning on up to 500K generated samples improves performance but remains insufficient to close the reasoning gap, positioning Video-MME-Logical as a scalable testbed for analyzing and improving temporal-logical reasoning in MLLMs.

Problem

• video temporal-logical reasoning
• multimodal large language models
• controlled benchmark
• temporal reasoning
• logical composition

Innovation

• temporal-logical reasoning
• controlled benchmark
• video reasoning
• multimodal large language models
• intermediate-state diagnostics