Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data

2026-06-24

AI Summary

This work addresses continuous-time mean-field control problems where only discrete-time transition data are available, proposing a model-free reinforcement learning approach. By embedding the discrete observations into the continuous-time Hamilton–Jacobi–Bellman (HJB) equation over Wasserstein space, the method preserves the generator structure and circumvents identifiability issues. Building upon this formulation, the authors derive one-step estimation-based policy evaluation and policy gradient theorems, leading to an Actor-Critic algorithm termed MF-PhiBE. Theoretically, the value function approximation error is shown to be of order Δt; notably, in the linear-quadratic setting, second-order accuracy is achieved using only single-step data. Empirical validation on both linear-quadratic regulator (LQR) and crowd-avoidance tasks demonstrates the efficacy of the proposed method.

Abstract

This paper addresses model-free continuous-time mean-field control in a setting where the population dynamics evolve continuously according to an unknown McKean-Vlasov stochastic differential equation, while only discrete-time transition data are available. In the model-based formulation, policy evaluation is naturally described by a stationary Hamilton-Jacobi-Bellman equation on $\mathcal P_2(\mathbb R^d)$, but this equation involves the drift and diffusion coefficients of the controlled McKean-Vlasov dynamics, which are not identifiable when only discrete-time data are available. On the other hand, a direct reduction to a time-discrete Bellman equation avoids the non-identifiability issue but loses the differential equation structure. To bridge these two viewpoints, we introduce a Mean-Field-PhiBE (MF-PhiBE), which incorporates discrete-time transition information into a continuous-time PDE on the Wasserstein space. The MF-PhiBE replaces the unknown infinitesimal drift and covariance in the policy-evaluation equation by one-step estimators computed from data, while preserving the generator structure of the McKean-Vlasov HJB equation. We also derive a policy-gradient theorem for entropy-regularized randomized feedback policies, expressing the actor direction through an action-wise infinitesimal advantage and the score of the policy. Combining these two ingredients yields a model-free actor-critic method. We prove a first-order consistency estimate showing that the value induced by an optimal MF-PhiBE policy approximates the optimal continuous-time value with an error of order $Δt$. In the linear-quadratic case, we show our approximation achieves second-order accuracy with only one-step data. Numerical experiments on an LQR benchmark and a crowd-aversion problem illustrate the proposed framework.

Problem

• mean-field control
• continuous-time reinforcement learning
• McKean-Vlasov dynamics
• discrete-time data
• model-free learning

Innovation

• Mean-Field Reinforcement Learning
• Continuous-Time Control
• McKean-Vlasov Dynamics
• PhiBE
• Wasserstein Space