Supervised Reward Inference

2025-02-25

AI Summary

This work addresses the limitation of reward inference methods that rely on strong assumptions about human behavior—such as optimality or predefined bias structures—by proposing a robust framework capable of inferring the true reward function from arbitrary human behavior, including suboptimal actions, errors, or goal-communicative demonstrations. Methodologically, it establishes supervised learning as a unified framework for reward inference for the first time, and theoretically proves its asymptotic Bayesian optimality under mild assumptions. The approach integrates Bayesian inference with supervised learning and is validated in robotic manipulation simulations: using only diverse suboptimal demonstrations, it efficiently and accurately recovers reward functions while exhibiting strong generalization. Crucially, the framework eliminates the need to pre-specify or model the underlying behavioral policy, thereby significantly enhancing practical applicability in real-world settings.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Learning & Optimization for ROB
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Existing approaches to reward inference from behavior typically assume that humans provide demonstrations according to specific models of behavior. However, humans often indicate their goals through a wide range of behaviors, from actions that are suboptimal due to poor planning or execution to behaviors which are intended to communicate goals rather than achieve them. We propose that supervised learning offers a unified framework to infer reward functions from any class of behavior, and show that such an approach is asymptotically Bayes-optimal under mild assumptions. Experiments on simulated robotic manipulation tasks show that our method can efficiently infer rewards from a wide variety of arbitrarily suboptimal demonstrations.

Problem

• Infer reward functions from diverse human behaviors
• Unify reward inference using supervised learning framework
• Handle suboptimal demonstrations in robotic manipulation tasks

Innovation

• Supervised learning for reward inference
• Unified framework for behavior analysis
• Efficient reward inference from demonstrations