🤖 AI Summary
To address the theoretical failure and performance degradation of Adversarial Inverse Reinforcement Learning (AIRL) in stochastic environments, this paper proposes a model-augmented AIRL framework that explicitly incorporates a dynamics model into the reward shaping process—yielding, for the first time, a model-driven reward design for AIRL in stochastic settings with rigorous theoretical guarantees. Our core contributions are: (1) a novel theoretical analysis framework establishing bounds on both reward estimation error and policy performance gap; and (2) an algorithm that jointly optimizes transition model estimation and adversarial training. Experiments on MuJoCo demonstrate that our method significantly outperforms existing baselines in stochastic environments, maintains competitive performance in deterministic ones, and achieves substantially improved sample efficiency.
📝 Abstract
In this paper, we aim to tackle the limitation of the Adversarial Inverse Reinforcement Learning (AIRL) method in stochastic environments where theoretical results cannot hold and performance is degraded. To address this issue, we propose a novel method which infuses the dynamics information into the reward shaping with the theoretical guarantee for the induced optimal policy in the stochastic environments. Incorporating our novel model-enhanced rewards, we present a novel Model-Enhanced AIRL framework, which integrates transition model estimation directly into reward shaping. Furthermore, we provide a comprehensive theoretical analysis of the reward error bound and performance difference bound for our method. The experimental results in MuJoCo benchmarks show that our method can achieve superior performance in stochastic environments and competitive performance in deterministic environments, with significant improvement in sample efficiency, compared to existing baselines.