Revealing Multimodal Causality with Large Language Models

2025-09-22

AI Summary

Addressing two key bottlenecks in causal discovery from multimodal unstructured data—insufficient intra- and inter-modal interaction modeling hindering causal variable identification, and structural ambiguity under observational data—the paper proposes the first end-to-end framework integrating multimodal large language models (MLLMs) with statistical causal inference. Methodologically, it introduces: (1) a contrastive factor discovery module that explicitly models cross-modal semantic interactions to localize causal variables; (2) statistical structure inference grounded in conditional independence testing; and (3) an iterative counterfactual reasoning mechanism to jointly resolve structural ambiguity. Evaluated on both synthetic and real-world multimodal datasets, the approach achieves significant improvements in causal factor identification accuracy and structural learning robustness. This work establishes a novel paradigm for trustworthy causal discovery empowered by MLLMs.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Computer Vision: Multi-modal Vision

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Uncovering cause-and-effect mechanisms from data is fundamental to scientific progress. While large language models (LLMs) show promise for enhancing causal discovery (CD) from unstructured data, their application to the increasingly prevalent multimodal setting remains a critical challenge. Even with the advent of multimodal LLMs (MLLMs), their efficacy in multimodal CD is hindered by two primary limitations: (1) difficulty in exploring intra- and inter-modal interactions for comprehensive causal variable identification; and (2) insufficiency to handle structural ambiguities with purely observational data. To address these challenges, we propose MLLM-CD, a novel framework for multimodal causal discovery from unstructured data. It consists of three key components: (1) a novel contrastive factor discovery module to identify genuine multimodal factors based on the interactions explored from contrastive sample pairs; (2) a statistical causal structure discovery module to infer causal relationships among discovered factors; and (3) an iterative multimodal counterfactual reasoning module to refine the discovery outcomes iteratively by incorporating the world knowledge and reasoning capabilities of MLLMs. Extensive experiments on both synthetic and real-world datasets demonstrate the effectiveness of MLLM-CD in revealing genuine factors and causal relationships among them from multimodal unstructured data.

Problem

• Addressing multimodal causal discovery challenges with large language models
• Identifying genuine multimodal factors through contrastive sample interactions
• Refining causal relationships using iterative counterfactual reasoning

Innovation

• Contrastive factor discovery module identifies multimodal factors
• Statistical causal structure discovery module infers relationships
• Iterative multimodal counterfactual reasoning module refines outcomes