MECD+: Unlocking Event-Level Causal Graph Discovery for Video Reasoning

2025-01-13

AI Summary

Existing methods struggle to model complex causal relationships among multiple events in long videos. This paper introduces the **Multi-Event Causal Discovery (MECD)** task and benchmark, the first to formally define and address automatic construction of **event-level causal graphs** from video clips and textual descriptions—revealing cross-temporal “why” and “how” causal mechanisms. Methodologically, we propose the **Event Granger Test framework**, which innovatively integrates front-door adjustment, counterfactual reasoning, and multi-hop contextual chain inference to mitigate confounding bias and spurious causality. We further design a Granger-inspired masked event prediction model with explicit video–language modality alignment. On the MECD benchmark, our approach outperforms GPT-4o and VideoChat2 by 5.77% and 2.70%, respectively, and the generated causal graphs significantly enhance downstream tasks—including video question answering and event prediction.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Computer Vision: Video Understanding & Activity Analysis

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Video causal reasoning aims to achieve a high-level understanding of videos from a causal perspective. However, it exhibits limitations in its scope, primarily executed in a question-answering paradigm and focusing on brief video segments containing isolated events and basic causal relations, lacking comprehensive and structured causality analysis for videos with multiple interconnected events. To fill this gap, we introduce a new task and dataset, Multi-Event Causal Discovery (MECD). It aims to uncover the causal relations between events distributed chronologically across long videos. Given visual segments and textual descriptions of events, MECD identifies the causal associations between these events to derive a comprehensive and structured event-level video causal graph explaining why and how the result event occurred. To address the challenges of MECD, we devise a novel framework inspired by the Granger Causality method, incorporating an efficient mask-based event prediction model to perform an Event Granger Test. It estimates causality by comparing the predicted result event when premise events are masked versus unmasked. Furthermore, we integrate causal inference techniques such as front-door adjustment and counterfactual inference to mitigate challenges in MECD like causality confounding and illusory causality. Additionally, context chain reasoning is introduced to conduct more robust and generalized reasoning. Experiments validate the effectiveness of our framework in reasoning complete causal relations, outperforming GPT-4o and VideoChat2 by 5.77% and 2.70%, respectively. Further experiments demonstrate that causal relation graphs can also contribute to downstream video understanding tasks such as video question answering and video event prediction.

Problem

• Causal Reasoning
• Complex Videos
• Multiple Correlated Events

Innovation

• Causal Relation Analysis
• Granger Causality
• Context Chain Reasoning