Learning Nonlinear Causal Reductions to Explain Reinforcement Learning Policies

2025-07-20

AI Summary

This work addresses the challenge of explaining reinforcement learning policies through causal attribution of success and failure. To tackle the difficulty of attributing causality in high-dimensional, nonlinear agent–environment interactions, we propose the first intervention-consistent nonlinear causal model reduction framework. It stimulates causal responses among states, actions, and rewards via randomized action perturbations, then jointly applies nonlinear dimensionality reduction and causal structure learning to automatically extract high-level behavioral patterns with explicit causal semantics from raw trajectories. We theoretically prove that, under generalized additive models, this framework admits a unique exact solution. Evaluated on benchmark tasks—including cart-pole control and robotic table tennis—the method successfully identifies policy biases, critical decision bottlenecks, and failure mechanisms, outperforming existing black-box explanation approaches by a significant margin.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Why do reinforcement learning (RL) policies fail or succeed? This is a challenging question due to the complex, high-dimensional nature of agent-environment interactions. In this work, we take a causal perspective on explaining the behavior of RL policies by viewing the states, actions, and rewards as variables in a low-level causal model. We introduce random perturbations to policy actions during execution and observe their effects on the cumulative reward, learning a simplified high-level causal model that explains these relationships. To this end, we develop a nonlinear Causal Model Reduction framework that ensures approximate interventional consistency, meaning the simplified high-level model responds to interventions in a similar way as the original complex system. We prove that for a class of nonlinear causal models, there exists a unique solution that achieves exact interventional consistency, ensuring learned explanations reflect meaningful causal patterns. Experiments on both synthetic causal models and practical RL tasks-including pendulum control and robot table tennis-demonstrate that our approach can uncover important behavioral patterns, biases, and failure modes in trained RL policies.

Problem

• Explain RL policy behavior via causal model reduction
• Learn high-level causal relationships from action perturbations
• Ensure interventional consistency in simplified causal explanations

Innovation

• Introduces random perturbations to policy actions
• Develops nonlinear Causal Model Reduction framework
• Ensures approximate interventional consistency in models