Using Deep Autoregressive Models as Causal Inference Engines

2024-09-27

arXiv.org

AI Summary

Existing causal inference methods are constrained by assumptions of low-dimensional, static confounders and single-action interventions, limiting their ability to model high-dimensional confounding and sequential interventions. This paper introduces the first general-purpose causal inference framework based on deep autoregressive models (e.g., Transformers). It employs a causal DAG-guided serialization scheme to encode structured causal data into token sequences, enabling unified estimation of intervention probabilities, counterfactual outcomes, and other causal quantities. Its core innovation is the first integration of the autoregressive paradigm into causal inference—supporting end-to-end, joint estimation of multiple causal targets within a single model, without reliance on low-dimensional or static confounding assumptions. Empirical evaluation across three diverse tasks—maze navigation, chess endgame solving, and academic keyword impact assessment—demonstrates substantial improvements in intervention prediction accuracy and causal reasoning efficiency.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Existing causal inference (CI) models are limited to primarily handling low-dimensional confounders and singleton actions. We propose an autoregressive (AR) CI framework capable of handling complex confounders and sequential actions common in modern applications. We accomplish this by {em sequencification}, transforming data from an underlying causal diagram into a sequence of tokens. This approach not only enables training with data generated from any DAG but also extends existing CI capabilities to accommodate estimating several statistical quantities using a {em single} model. We can directly predict interventional probabilities, simplifying inference and enhancing outcome prediction accuracy. We demonstrate that an AR model adapted for CI is efficient and effective in various complex applications such as navigating mazes, playing chess endgames, and evaluating the impact of certain keywords on paper acceptance rates.

Problem

• Handling complex confounders in causal inference
• Extending CI to sequential actions and data
• Estimating multiple causal quantities with one model

Innovation

• Autoregressive framework for complex confounders
• Sequencification transforms data into tokens
• Single model estimates multiple causal quantities