Cost Function Estimation Using Inverse Reinforcement Learning with Minimal Observations

2025-05-13

AI Summary

This work addresses the problem of inferring an optimal cost function in continuous-state spaces from extremely sparse expert demonstrations. We propose an iterative inverse reinforcement learning (IRL) framework grounded in the maximum-entropy principle. Methodologically, it integrates stepwise observation weighting with differentiation, adaptive step-size scheduling to preserve feature consistency, and optimal control-based generation of high-informativeness trajectories. Joint iterative optimization of cost weights and the cost function eliminates reliance on large-scale demonstration data. Our key contribution is the first incorporation of dynamic step-size selection and feature-similarity constraints into a maximum-entropy IRL framework, substantially improving estimation robustness and accuracy under sparse demonstrations. Evaluated across multiple continuous-control benchmark environments, our method achieves 30–50% faster convergence than two state-of-the-art baselines and attains superior estimation accuracy using only 1–3 expert trajectories.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Sampling/Simulation-based Search
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

We present an iterative inverse reinforcement learning algorithm to infer optimal cost functions in continuous spaces. Based on a popular maximum entropy criteria, our approach iteratively finds a weight improvement step and proposes a method to find an appropriate step size that ensures learned cost function features remain similar to the demonstrated trajectory features. In contrast to similar approaches, our algorithm can individually tune the effectiveness of each observation for the partition function and does not need a large sample set, enabling faster learning. We generate sample trajectories by solving an optimal control problem instead of random sampling, leading to more informative trajectories. The performance of our method is compared to two state of the art algorithms to demonstrate its benefits in several simulated environments.

Problem

• Estimating cost functions in continuous spaces using inverse reinforcement learning
• Improving learning efficiency with minimal observation samples
• Generating informative trajectories via optimal control problem solving

Innovation

• Iterative inverse reinforcement learning in continuous spaces
• Optimal control for generating informative trajectories
• Individual observation tuning for faster learning