BiCQL-ML: A Bi-Level Conservative Q-Learning Framework for Maximum Likelihood Inverse Reinforcement Learning

2025-11-27

AI Summary

This paper addresses the offline inverse reinforcement learning (IRL) problem—recovering an interpretable reward function solely from static expert demonstrations, without online interaction. We propose an off-policy, bilevel optimization framework: the upper level performs maximum-likelihood reward estimation via soft value matching, while the lower level implicitly models expert behavior using conservative Q-learning (CQL), jointly optimizing the reward function and a conservative Q-function. We theoretically establish convergence to the soft-optimal reward function. Empirically, our method achieves significantly improved reward recovery accuracy and downstream policy performance on standard offline RL benchmarks, outperforming existing IRL and offline imitation learning approaches. The key innovation lies in eliminating explicit policy learning; instead, we employ an end-to-end bilevel structure that enables more robust and precise reward inference.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Offline inverse reinforcement learning (IRL) aims to recover a reward function that explains expert behavior using only fixed demonstration data, without any additional online interaction. We propose BiCQL-ML, a policy-free offline IRL algorithm that jointly optimizes a reward function and a conservative Q-function in a bi-level framework, thereby avoiding explicit policy learning. The method alternates between (i) learning a conservative Q-function via Conservative Q-Learning (CQL) under the current reward, and (ii) updating the reward parameters to maximize the expected Q-values of expert actions while suppressing over-generalization to out-of-distribution actions. This procedure can be viewed as maximum likelihood estimation under a soft value matching principle. We provide theoretical guarantees that BiCQL-ML converges to a reward function under which the expert policy is soft-optimal. Empirically, we show on standard offline RL benchmarks that BiCQL-ML improves both reward recovery and downstream policy performance compared to existing offline IRL baselines.

Problem

• Recover reward functions from fixed expert data without online interaction.
• Avoid explicit policy learning by jointly optimizing reward and Q-functions.
• Suppress over-generalization to out-of-distribution actions in reward recovery.

Innovation

• Bi-level framework jointly optimizes reward and Q-function
• Uses Conservative Q-Learning to suppress out-of-distribution actions
• Performs maximum likelihood estimation via soft value matching