Exploring Multi-Modal Data with Tool-Augmented LLM Agents for Precise Causal Discovery

2024-12-18

AI Summary

Traditional causal discovery methods rely solely on observational data, neglecting semantic cues and struggling with multimodal information. To address this, we propose MATMCD—the first tool-augmented, multi-agent large language model framework for causal discovery—designed to enable knowledge-guided causal graph construction by decoupling data augmentation from causal-constrained reasoning and integrating multimodal semantic cues (text, images, time series). Our method synergizes cross-modal retrieval and alignment, symbolic causal graph modeling, and tool-augmented reasoning. Evaluated on seven benchmark datasets, MATMCD achieves an average 23.6% improvement in causal graph accuracy over state-of-the-art unimodal and LLM-based baselines. These results empirically validate the effectiveness and generalizability of a multimodal semantic–driven paradigm for causal discovery.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Causal discovery is an imperative foundation for decision-making across domains, such as smart health, AI for drug discovery and AIOps. Traditional statistical causal discovery methods, while well-established, predominantly rely on observational data and often overlook the semantic cues inherent in cause-and-effect relationships. The advent of Large Language Models (LLMs) has ushered in an affordable way of leveraging the semantic cues for knowledge-driven causal discovery, but the development of LLMs for causal discovery lags behind other areas, particularly in the exploration of multi-modal data. To bridge the gap, we introduce MATMCD, a multi-agent system powered by tool-augmented LLMs. MATMCD has two key agents: a Data Augmentation agent that retrieves and processes modality-augmented data, and a Causal Constraint agent that integrates multi-modal data for knowledge-driven reasoning. The proposed design of the inner-workings ensures successful cooperation of the agents. Our empirical study across seven datasets suggests the significant potential of multi-modality enhanced causal discovery.

Problem

• Bridging the gap in multi-modal data exploration for causal discovery
• Enhancing causal discovery with semantic cues using LLM agents
• Overcoming limitations of traditional statistical causal discovery methods

Innovation

• Tool-augmented LLMs for multi-modal data
• Data Augmentation agent processes modality-augmented data
• Causal Constraint agent integrates multi-modal reasoning