Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learning

2026-06-20

AI Summary

Although adversarial imitation learning (AIL) outperforms behavioral cloning (BC), it relies heavily on extensive online interactions and lacks a theoretical understanding of pretraining mechanisms. This work identifies reward estimation error as the primary cause of AIL’s suboptimality and establishes, for the first time, theoretical guarantees for pretraining in AIL. The authors propose CoPT-AIL, a joint pretraining framework that simultaneously initializes both the policy and reward function through a single BC step, revealing an intrinsic connection between expert policies and shaping rewards. Theoretical analysis demonstrates that CoPT-AIL yields a tighter bound on the imitation gap, and empirical results confirm its significant superiority over existing AIL methods.

Abstract

Adversarial imitation learning (AIL) achieves high-quality imitation compared to behavioral cloning (BC), but demands substantial online environment interaction. Recent empirical work has explored initializing AIL algorithms with BC pretrained policies to address this limitation, yet a rigorous theoretical understanding of pretraining's role in AIL remains elusive. This paper provides a systematic theoretical analysis and introduces principled pretraining algorithms for accelerating AIL. We begin by analyzing AIL with policy pretraining alone, identifying reward error as the dominant source of suboptimality. This reveals a critical and previously overlooked gap: the absence of reward pretraining. Motivated by this finding, we develop a principled policy-reward co-pretraining approach grounded in a reward shaping analysis. Our analysis uncovers a fundamental connection between expert policies and shaping rewards, which naturally gives rise to CoPT-AIL, an approach that jointly pretrains both policy and reward through a single BC procedure. We prove that CoPT-AIL achieves an improved imitation gap bound over standard AIL, establishing the first theoretical guarantee for the benefits of pretraining in AIL. Experimental results confirm CoPT-AIL's superior performance over existing AIL methods.

Problem

• Adversarial Imitation Learning
• Policy Pretraining
• Reward Pretraining
• Imitation Gap
• Sample Efficiency

Innovation

• policy-reward co-pretraining
• adversarial imitation learning
• reward shaping
• theoretical guarantee
• imitation gap