Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies

2026-07-12

AI Summary

This work proposes a model-free reinforcement learning approach for continuous-time extended mean-field control problems where the joint distribution of states and controls exhibits explicit dependence. By adopting deterministic feedback policies, the state-action distribution is treated as the pushforward of the state distribution, thereby circumventing optimization over stochastic kernels. The study derives, for the first time, a policy gradient formula in Wasserstein space that incorporates both action derivatives and measure derivatives with respect to the control distribution. A martingale-driven learning framework is developed, integrating particle approximation, measure-dependent neural networks, temporal-difference learning, and exploration mechanisms to yield a continuous-time deep deterministic policy gradient algorithm tailored to this class of problems. The method demonstrates superior efficiency, stability, and robustness in applications including Cucker–Smale consensus control and optimal liquidation under trade crowding.

Abstract

This paper develops a model-free reinforcement learning framework for continuous--time extended mean field control problems, where both the dynamics and reward may depend on the joint distribution of states and controls. We adopt deterministic feedback policies, under which the state--action distribution is induced directly as a push--forward of the state law. This avoids optimization over stochastic kernels and bypasses key limitations of existing approaches in extended mean field settings. We first establish a model--free sensitivity formula for parameterized McKean--Vlasov dynamics and use it to derive a deterministic policy gradient formula expressed through an advantage--rate function on the Wasserstein space. We then refine this formula by introducing local value and advantage--rate representations that depend on the state, action, and joint state--action distribution, yielding a policy gradient that includes both action derivatives and measure--derivative terms with respect to the control distribution. These characterizations lead to a martingale--based learning principle and motivate a continuous--time deep deterministic policy gradient algorithm combining particle approximations, measure--dependent neural networks, temporal--difference learning, and exploration in either action or parameter space. Numerical experiments on stochastic Cucker--Smale consensus control and optimal liquidation with trade crowding demonstrate the efficiency, stability, and robustness of the proposed method, including problems with explicit dependence on the control distribution.

Problem

• extended mean field control
• deterministic policies
• reinforcement learning
• McKean–Vlasov dynamics
• Wasserstein space

Innovation

• Extended Mean Field Control
• Deterministic Policy Gradient
• Wasserstein Space
• McKean–Vlasov Dynamics
• Measure-Dependent Neural Networks