Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward

2026-06-15

AI Summary

This study addresses the problem of recovering unknown reward functions from expert demonstrations in mean-field games under the average-reward criterion. To overcome the non-contraction of the Bellman operator caused by the absence of a discount factor, the authors introduce a sub-stochastic kernel based on minorization conditions, which ensures strict contraction of the soft Bellman operator and establishes Fréchet differentiability and smoothness of the log-likelihood score. Within the occupancy measure framework, the approach unifies the treatment of two classes of reward functions by integrating convex duality reformulation, RKHS-based reward modeling, and gradient ascent algorithms. Experiments on malware propagation and consumer choice models demonstrate that the recovered policies closely match expert behavior, validating the method’s effectiveness and convergence properties.

Abstract

We study inverse reinforcement learning for discrete-time, infinite-horizon mean-field games (MFGs) under an average-reward criterion. Expert demonstrations are assumed to arise from a stationary mean-field equilibrium under an unknown reward, and the goal is to recover a policy explaining the observed behaviour via the maximum causal entropy principle. We formulate the inverse problem by enforcing consistency with the expert mean-field term and long-run feature expectations, treating two reward classes within a unified occupation-measure framework. For finite-dimensional linear rewards, we give a convex dual reformulation with an explicit log-partition objective, and prove smoothness and curvature properties justifying constant-step-size gradient descent. For infinite-dimensional RKHS rewards, we develop a Lagrangian relaxation whose inner-maximising policy is characterised by a soft Bellman equation. The main obstacle is the absence of a discount-factor contraction. We resolve this by introducing a minorisation-based sub-stochastic kernel that yields a strict contraction of the soft Bellman operator. We establish Fréchet differentiability and Lipschitz smoothness of the log-likelihood score, leading to a gradient ascent algorithm with convergence guarantees. Two numerical examples, a malware-spread MFG and an RKHS-based consumer-choice model, show that the recovered policies closely match expert behaviour.

Problem

• inverse reinforcement learning
• mean-field games
• average reward
• maximum entropy
• expert demonstrations

Innovation

• Maximum Entropy Inverse Reinforcement Learning
• Mean-Field Games
• Average Reward Criterion
• Soft Bellman Equation
• Reproducing Kernel Hilbert Space (RKHS)