Confounded Causal Imitation Learning with Instrumental Variables

2025-07-23

AI Summary

To address policy estimation bias induced by unobserved confounders in imitation learning, this paper proposes a causal imitation learning framework grounded in instrumental variable (IV) methods. The framework decouples confounding effects across time steps and introduces a two-stage policy learning algorithm: the first stage establishes necessary and sufficient identification conditions for valid IVs via a pseudo-variable testing criterion; the second stage achieves unbiased policy estimation under both simulation and offline settings. Integrating causal inference, two-stage regression, and verifiable IV construction, the method significantly enhances IV identifiability robustness and policy generalization capability. Experiments on multiple benchmark tasks demonstrate that the proposed approach outperforms existing imitation learning methods, effectively mitigating confounding bias while ensuring policy consistency.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Intelligent Robots: State Estimation
• Reasoning under Uncertainty: Causality

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Imitation learning from demonstrations usually suffers from the confounding effects of unmeasured variables (i.e., unmeasured confounders) on the states and actions. If ignoring them, a biased estimation of the policy would be entailed. To break up this confounding gap, in this paper, we take the best of the strong power of instrumental variables (IV) and propose a Confounded Causal Imitation Learning (C2L) model. This model accommodates confounders that influence actions across multiple timesteps, rather than being restricted to immediate temporal dependencies. We develop a two-stage imitation learning framework for valid IV identification and policy optimization. In particular, in the first stage, we construct a testing criterion based on the defined pseudo-variable, with which we achieve identifying a valid IV for the C2L models. Such a criterion entails the sufficient and necessary identifiability conditions for IV validity. In the second stage, with the identified IV, we propose two candidate policy learning approaches: one is based on a simulator, while the other is offline. Extensive experiments verified the effectiveness of identifying the valid IV as well as learning the policy.

Problem

• Addresses confounding effects of unmeasured variables in imitation learning
• Proposes instrumental variables to break confounding gaps in policy estimation
• Develops two-stage framework for IV identification and policy optimization

Innovation

• Uses instrumental variables for confounder removal
• Two-stage framework for IV identification and policy optimization
• Offers both simulator-based and offline policy learning