Near-Optimal Second-Order Guarantees for Model-Based Adversarial Imitation Learning

2025-10-10

AI Summary

This work addresses online adversarial imitation learning (AIL), focusing on efficiently leveraging offline expert demonstrations alongside online environment interactions—without access to reward signals. To this end, we propose MB-AIL, the first model-based AIL algorithm, which integrates learned dynamics models, second-order error analysis, and information-theoretic lower-bound construction. Theoretically, we establish the first horizon-free upper bound on second-order sample complexity, proving minimax optimality in both expert demonstration size and online interaction budget. This result holds under general function approximation assumptions, yielding the strongest theoretical guarantees for AIL to date. Empirically, MB-AIL achieves significantly higher sample efficiency than or matches state-of-the-art model-free AIL methods across diverse benchmarks.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Multiagent Systems: Adversarial Agents
• Search and Optimization: Adversarial Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We study online adversarial imitation learning (AIL), where an agent learns from offline expert demonstrations and interacts with the environment online without access to rewards. Despite strong empirical results, the benefits of online interaction and the impact of stochasticity remain poorly understood. We address these gaps by introducing a model-based AIL algorithm (MB-AIL) and establish its horizon-free, second-order sample-complexity guarantees under general function approximations for both expert data and reward-free interactions. These second-order bounds provide an instance-dependent result that can scale with the variance of returns under the relevant policies and therefore tighten as the system approaches determinism. Together with second-order, information-theoretic lower bounds on a newly constructed hard-instance family, we show that MB-AIL attains minimax-optimal sample complexity for online interaction (up to logarithmic factors) with limited expert demonstrations and matches the lower bound for expert demonstrations in terms of the dependence on horizon $H$, precision $ε$ and the policy variance $σ^2$. Experiments further validate our theoretical findings and demonstrate that a practical implementation of MB-AIL matches or surpasses the sample efficiency of existing methods.

Problem

• Develops model-based adversarial imitation learning from offline demonstrations
• Establishes second-order sample complexity guarantees under function approximation
• Achieves minimax-optimal sample efficiency for online reward-free interactions

Innovation

• Model-based adversarial imitation learning algorithm
• Horizon-free second-order sample complexity guarantees
• Minimax-optimal sample complexity with limited demonstrations