Reward Compatibility: A Framework for Inverse RL

2025-01-14

AI Summary

This paper addresses the inaccuracy in expert reward function inference and limited theoretical applicability of inverse reinforcement learning (IRL). We propose a reward compatibility framework that introduces, for the first time, a continuous compatibility metric—replacing traditional binary feasibility judgments—to uniformly model both optimal and suboptimal demonstrations, as well as online and offline data settings. Compatibility is defined via policy performance deviation, enabling principled integration of demonstration quality and uncertainty. We design a scalable optimization algorithm and derive a theoretically grounded upper bound on sample complexity. Our framework achieves provably efficient IRL across tabular to large-scale MDPs, significantly expanding the theoretical scope of IRL: it accommodates imperfect demonstrations, provides tight sample complexity guarantees, and balances theoretical rigor with computational tractability.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

We provide an original theoretical study of Inverse Reinforcement Learning (IRL) through the lens of reward compatibility, a novel framework to quantify the compatibility of a reward with the given expert's demonstrations. Intuitively, a reward is more compatible with the demonstrations the closer the performance of the expert's policy computed with that reward is to the optimal performance for that reward. This generalizes the notion of feasible reward set, the most common framework in the theoretical IRL literature, for which a reward is either compatible or not compatible. The grayscale introduced by the reward compatibility is the key to extend the realm of provably efficient IRL far beyond what is attainable with the feasible reward set: from tabular to large-scale MDPs. We analyze the IRL problem across various settings, including optimal and suboptimal expert's demonstrations and both online and offline data collection. For all of these dimensions, we provide a tractable algorithm and corresponding sample complexity analysis, as well as various insights on reward compatibility and how the framework can pave the way to yet more general problem settings.

Problem

• Complex Decision-making
• Inverse Reinforcement Learning
• Expert Behavior

Innovation

• Reward Compatibility
• Inverse Reinforcement Learning
• Fine-grained Matching