Inverse Reinforcement Learning Using Just Classification and a Few Regressions

2025-09-25

AI Summary

Inverse reinforcement learning (IRL) aims to infer an underlying reward function from expert demonstrations, yet existing maximum-entropy IRL methods rely on inner-loop optimization, dynamic programming, or adversarial training—making them ill-suited for modern function approximators. This paper proposes a modular framework that reformulates maximum-likelihood IRL as a linear fixed-point equation in the space of behavior policies. The problem is thus decoupled into two standard supervised learning subtasks: probabilistic classification for policy estimation and iterative regression for fixed-point solving. The approach naturally accommodates expressive models such as deep neural networks and gradient-boosted trees. Theoretically, we provide an exact characterization of the optimal solution and derive finite-sample error bounds. Empirically, our method matches or surpasses classical MaxEnt IRL across benchmark tasks, demonstrating improved scalability and compatibility with contemporary machine learning infrastructure.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Inverse reinforcement learning (IRL) aims to explain observed behavior by uncovering an underlying reward. In the maximum-entropy or Gumbel-shocks-to-reward frameworks, this amounts to fitting a reward function and a soft value function that together satisfy the soft Bellman consistency condition and maximize the likelihood of observed actions. While this perspective has had enormous impact in imitation learning for robotics and understanding dynamic choices in economics, practical learning algorithms often involve delicate inner-loop optimization, repeated dynamic programming, or adversarial training, all of which complicate the use of modern, highly expressive function approximators like neural nets and boosting. We revisit softmax IRL and show that the population maximum-likelihood solution is characterized by a linear fixed-point equation involving the behavior policy. This observation reduces IRL to two off-the-shelf supervised learning problems: probabilistic classification to estimate the behavior policy, and iterative regression to solve the fixed point. The resulting method is simple and modular across function approximation classes and algorithms. We provide a precise characterization of the optimal solution, a generic oracle-based algorithm, finite-sample error bounds, and empirical results showing competitive or superior performance to MaxEnt IRL.

Problem

• Inverse reinforcement learning explains behavior by uncovering rewards
• Existing methods involve complex optimization and dynamic programming
• New approach simplifies IRL using classification and regression techniques

Innovation

• Uses probabilistic classification for policy estimation
• Solves linear fixed-point equation via iterative regression
• Replaces complex optimization with modular supervised learning