Understanding Inverse Reinforcement Learning under Overparameterization: Non-Asymptotic Analysis and Global Optimality

2025-03-22

AI Summary

This work addresses two fundamental challenges in inverse reinforcement learning (IRL): the inability of conventional methods to model reward functions nonlinearly and their poor computational efficiency. We propose the first single-loop, two-timescale IRL algorithm capable of learning neural-network-parameterized reward functions. Departing from the standard linear reward assumption and nested optimization paradigm, our method operates under overparameterized neural network settings and leverages neural tangent kernel (NTK) theory, nonconvex–nonconcave saddle-point analysis, and two-timescale stochastic gradient updates. We establish a non-asymptotic convergence guarantee to the globally optimal reward function and policy. To our knowledge, this is the first theoretical result providing global optimality and learnability guarantees for deep IRL beyond linear reward structures—significantly enhancing both computational feasibility in high-dimensional settings and theoretical completeness.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The goal of the Inverse reinforcement learning (IRL) task is to identify the underlying reward function and the corresponding optimal policy from a set of expert demonstrations. While most IRL algorithms' theoretical guarantees rely on a linear reward structure, we aim to extend the theoretical understanding of IRL to scenarios where the reward function is parameterized by neural networks. Meanwhile, conventional IRL algorithms usually adopt a nested structure, leading to computational inefficiency, especially in high-dimensional settings. To address this problem, we propose the first two-timescale single-loop IRL algorithm under neural network parameterized reward and provide a non-asymptotic convergence analysis under overparameterization. Although prior optimality results for linear rewards do not apply, we show that our algorithm can identify the globally optimal reward and policy under certain neural network structures. This is the first IRL algorithm with a non-asymptotic convergence guarantee that provably achieves global optimality in neural network settings.

Problem

• Extends IRL theory to neural network reward functions
• Addresses computational inefficiency in high-dimensional IRL
• Ensures global optimality in neural network IRL settings

Innovation

• Neural network parameterized reward function
• Two-timescale single-loop IRL algorithm
• Non-asymptotic convergence global optimality