DMC-CF: Dynamic Multimodal CounterFactual QA benchmark for Causal Reasoning

2026-05-28

AI Summary

This work addresses the absence of large-scale, real-world multimodal benchmarks for causal counterfactual reasoning, which hinders effective evaluation of multimodal large language models’ (MLLMs) causal understanding. To bridge this gap, the authors introduce DMC-CF-Static, the first static benchmark constructed from real-world videos, and further propose a Dynamic Graph Intervention (DGI) framework to extend it into DMC-CF-Dynamic—a dynamic, contamination-resistant evaluation benchmark. By integrating causal graph modeling, dynamic intervention mechanisms, and multimodal counterfactual question generation, this benchmark enables more realistic assessment of causal reasoning capabilities. Experimental results reveal that current MLLMs perform substantially worse on such tasks, underscoring the necessity of this benchmark for rigorous model evaluation and future development.

Abstract

With the rapid advancement of multimodal large language models (MLLMs), models have demonstrated increasingly powerful multimodal capabilities. However, whether MLLMs trained through statistical learning can truly understand the causal relationships underlying the real world remains a key research question. In recent years, numerous multimodal causal reasoning datasets have been proposed. Nevertheless, these datasets are either limited in scale or constructed from synthetic images and videos, cartoon-based content, or other non-realistic multimodal sources. To address these limitations, we collect real-world videos and construct DMC-CF-Static, a large-scale benchmark for multimodal causal counterfactual reasoning. Furthermore, to mitigate issues such as data contamination in traditional static evaluation, we represent causal events using causal graphs and propose the Dynamic Graph Intervention (DGI) framework to build the dynamic evaluation benchmark DMC-CF-Dynamic from DMC-CF-Static. Experimental results on the overall DMC-CF, which includes both static and dynamic evaluation benchmarks, demonstrate that the multimodal causal reasoning capabilities of current multimodal large language models in real-world scenarios still require substantial improvement.

Problem

• multimodal causal reasoning
• counterfactual QA
• real-world videos
• causal understanding
• benchmark

Innovation

• multimodal causal reasoning
• counterfactual QA
• real-world video benchmark
• dynamic graph intervention
• causal graph