Robust $Q$-learning for mean-field control under Wasserstein uncertainty in common noise

2026-06-18

AI Summary

This work addresses discrete-time mean-field control problems under common noise with Wasserstein distributional uncertainty by proposing a novel robust Q-learning algorithm. The method uniquely integrates Wasserstein duality reformulation into the mean-field Q-learning framework and combines a quantization-projection mechanism with asynchronous update strategies to effectively handle distributional ambiguity. Theoretical analysis establishes finite-time convergence guarantees for both synchronous and asynchronous learning schemes, with efficient solutions achieved through Bellman iteration approximation. Experimental validation on systemic risk and epidemic models demonstrates the algorithm’s convergence, robustness, and the inherent trade-off between robustness and performance, highlighting its practical utility and methodological innovation.

Abstract

In this article, we present a robust $Q$-learning algorithm for discrete-time mean-field control problems under Wasserstein uncertainty in the common noise law. The algorithm combines a quantization-and-projection scheme with a Wasserstein dual reformulation on the common-noise space. We establish its convergence together with finite-time iteration bounds for both synchronous and asynchronous learning schemes. Numerical experiments on systemic risk and epidemic models compare the asynchronous implementation with an idealized Bellman iteration, illustrate the robustness-performance tradeoff under common-noise misspecification, and report the observed convergence behavior of the asynchronous $Q$-learning algorithm.

Problem

• Robust Q-learning
• Mean-field control
• Wasserstein uncertainty
• Common noise
• Reinforcement learning

Innovation

• Robust Q-learning
• Mean-field control
• Wasserstein uncertainty
• Common noise
• Quantization-and-projection