Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms

2026-04-29

AI Summary

This study addresses the absence of implementable Q-learning algorithms for continuous-time mean-field control problems involving controlled common noise. Building upon a relaxed control framework, the authors establish a dynamic relationship between the value function and the Q-function using martingale orthogonality conditions, and approximate unobservable quantities via observable exploratory data. They innovatively propose an Actor-Critic Q-learning algorithm: in the Critic step, function estimates are updated leveraging the martingale condition, while the Actor step iteratively optimizes the policy. This work presents the first Q-learning approach based solely on observable data for this class of problems and proves the convergence of the Actor iteration in the linear-quadratic (LQ) infinite-horizon setting. Numerical experiments on both LQ and non-LQ examples demonstrate the algorithm’s effectiveness and broad applicability.

Abstract

This paper is a continuation work of Ren et al. (2026) aiming to further devise q-learning algorithms for mean-field control (MFC) with controlled common noise. Based on the relaxed control formulation, we first establish the martingale condition of the value function and the Iq-function by evaluating along the conditional state distributions generated by all test policies. As the data in the relaxed control formulation are not observable in practice, we quantify the error incurred when they are replaced by the observable ones in the exploratory formulation under discretely sampled actions. This, together with a two-layer fixed point characterization of an optimal policy in Ren et al. (2026), allows us to propose several algorithms including the Actor-Critic q-learning algorithm, in which the policy is updated in the Actor-step based on the iteration rule induced by the improved Iq-function, and the value function and Iq-function are updated in the Critic-step based on the martingale orthogonality condition using the data from the exploratory formulation. We also establish the convergence of the inner iterations in the Actor-step in an infinite-horizon linear quadratic (LQ) framework. In two examples, within and beyond LQ framework, our q-learning algorithms are implemented with satisfactory performance.

Problem

• mean-field control
• common noise
• q-learning
• continuous-time
• reinforcement learning

Innovation

• mean-field control
• continuous-time q-learning
• common noise
• actor-critic algorithm
• martingale condition