MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning

2024-09-26

AI Summary

Existing video analysis methods struggle to model complex causal relationships among multiple events in long videos, particularly in identifying the causes of event occurrences. To address this, we introduce MECD—the first structured multi-event causal discovery task—and its corresponding benchmark dataset, designed to automatically construct event-level causal graphs from video clips and textual descriptions. Methodologically, we propose the first event-level Granger causality framework, integrating front-door adjustment and counterfactual reasoning to mitigate confounding bias and spurious correlations. Additionally, we design a mask-based multimodal event prediction model that enables video–language aligned causal inference. On the MECD benchmark, our approach achieves significant accuracy gains over strong baselines: +5.7% over GPT-4o and +4.1% over VideoLLaVA. These results demonstrate the effectiveness and advancement of our multi-event causal modeling framework for video understanding.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Video causal reasoning aims to achieve a high-level understanding of video content from a causal perspective. However, current video reasoning tasks are limited in scope, primarily executed in a question-answering paradigm and focusing on short videos containing only a single event and simple causal relationships, lacking comprehensive and structured causality analysis for videos with multiple events. To fill this gap, we introduce a new task and dataset, Multi-Event Causal Discovery (MECD). It aims to uncover the causal relationships between events distributed chronologically across long videos. Given visual segments and textual descriptions of events, MECD requires identifying the causal associations between these events to derive a comprehensive, structured event-level video causal diagram explaining why and how the final result event occurred. To address MECD, we devise a novel framework inspired by the Granger Causality method, using an efficient mask-based event prediction model to perform an Event Granger Test, which estimates causality by comparing the predicted result event when premise events are masked versus unmasked. Furthermore, we integrate causal inference techniques such as front-door adjustment and counterfactual inference to address challenges in MECD like causality confounding and illusory causality. Experiments validate the effectiveness of our framework in providing causal relationships in multi-event videos, outperforming GPT-4o and VideoLLaVA by 5.7% and 4.1%, respectively.

Problem

• Complex Causal Relationships
• Long Video Analysis
• Event Understanding

Innovation

• Multi-event Causal Discovery
• Event Granger Test
• Counterfactual Inference