A Graphical Approach to State Variable Selection in Off-policy Learning

2025-01-01

AI Summary

Off-policy learning from non-randomized historical data faces challenges in aligning state variable selection with causal identification, particularly in general decision-making processes such as dynamic treatment regimes (DTRs) and Markov decision processes (MDPs). Method: We propose the first unified graphical identifiability criterion based on acyclic directed mixed graphs (ADMGs), integrating causal inference principles with ADMG modeling to bridge theoretical gaps between dynamic medical interventions and offline reinforcement learning, while clarifying widespread misuses of causal assumptions. Contribution/Results: Evaluated on a dynamic pricing simulation in container logistics, our criterion demonstrates that violating it leads to suboptimal policies; conversely, an evaluation framework built upon it significantly improves the causal validity and reliability of offline policy estimation. The approach provides an interpretable, verifiable causal foundation for real-world applications.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation

Abstract

Sequential decision problems are widely studied across many areas of science. A key challenge when learning policies from historical data - a practice commonly referred to as off-policy learning - is how to ``identify'' the impact of a policy of interest when the observed data are not randomized. Off-policy learning has mainly been studied in two settings: dynamic treatment regimes (DTRs), where the focus is on controlling confounding in medical problems with short decision horizons, and offline reinforcement learning (RL), where the focus is on dimension reduction in closed systems such as games. The gap between these two well studied settings has limited the wider application of off-policy learning to many real-world problems. Using the theory for causal inference based on acyclic directed mixed graph (ADMGs), we provide a set of graphical identification criteria in general decision processes that encompass both DTRs and MDPs. We discuss how our results relate to the often implicit causal assumptions made in the DTR and RL literatures and further clarify several common misconceptions. Finally, we present a realistic simulation study for the dynamic pricing problem encountered in container logistics, and demonstrate how violations of our graphical criteria can lead to suboptimal policies.

Problem

• Off-policy Learning
• Non-random Data
• Real-world Applications

Innovation

• ADMGs (Acyclic Directed Mixed Graphs)
• Off-policy Learning
• Decision Optimization in Healthcare and Gaming