Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations

2026-04-29

AI Summary

This work addresses the lack of continuous-time Q-learning theory for entropy-regularized mean-field control problems with common noise. By adopting a relaxed control framework, the authors formulate an exploratory Hamilton–Jacobi–Bellman (HJB) equation and introduce an integrated q-function (Iq-function), proving that the optimal policy can be characterized as a bilevel fixed point of its argmax operator. This approach effectively circumvents the policy iteration complexities induced by common noise. In discrete action spaces, the exploratory value function is shown to converge to the relaxed control value function, and the existence and uniqueness of an optimal one-step policy iteration are established. Notably, in the linear–quadratic (LQ) setting, the optimal policy is explicitly derived in Gaussian form, thereby laying the first theoretical foundation for continuous-time mean-field Q-learning with common noise.

Abstract

This paper investigates the continuous-time counterpart of the Q-function for entropy-regularized mean-field control (MFC) with controlled common noise, coined as q-function by Jia and Zhou (2023) in the single agent's model. We first show that, under discretely sampled actions, the value function in the exploratory formulation converges to the one in the relaxed control formulation as the time grid refines. Leveraging the relaxed control formulation, we derive the exploratory Hamilton-Jacobi-Bellman (HJB) equation, in which the controlled common noise gives rise to an additional nonlinear functional of policy, rendering the policy iteration intricate. Under certain concavity condition, we establish the existence and uniqueness of the optimal one-step policy iteration via a first-order condition using the partial linear functional derivative with respect to policy. The policy improvement at each iteration is verified by relating to an entropy-regularized optimization problem over the space of policies. In the mean-field setting, we introduce the integrated q-function (Iq-function) defined on the state distribution and the policy, and it is shown that an optimal policy is identified as a two-layer fixed point to the argmax operator of the Iq-function. Finally, we provide the explicit characterization of an optimal policy as a Gaussian distribution in the general linear-quadratic (LQ) setting.

Problem

• mean-field control
• common noise
• continuous-time q-learning
• entropy regularization
• relaxed control

Innovation

• mean-field control
• continuous-time q-learning
• common noise
• exploratory HJB equation
• integrated q-function