Reinforcement Learning for Finite Space Mean-Field Type Games

2024-09-25

arXiv.org

AI Summary

Addressing the computational intractability and poor scalability of Nash equilibrium computation for mean-field games (MFTGs) with finite state spaces in large-scale coalition games, this paper proposes the first reinforcement learning framework for MFTGs that simultaneously provides theoretical guarantees and high-dimensional scalability. We establish a rigorous theoretical connection between MFTG solutions and approximate Nash equilibria in finite-size coalition games. Methodologically, we introduce a dual-path algorithm: (i) “quantization + Nash Q-learning”, ensuring convergence under mild conditions, and (ii) a deep reinforcement learning variant, enabling adaptation to high-dimensional mean-field distributions. By integrating mean-field space quantization, stability analysis, and extensive evaluation across five benchmark environments, our approach successfully handles mean-field distributions of dimension up to 200—surpassing existing methods in both computational efficiency and scalability.

Technology Category

• Multiagent Systems: Mechanism Design
• Game Theory and Economic Paradigms: Cooperative Game Theory
• Machine Learning: Reinforcement Learning

Application Category

• Economics, Online Markets and Human Computation: Social networks and social learning
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Mean field type games (MFTGs) describe Nash equilibria between large coalitions: each coalition consists of a continuum of cooperative agents who maximize the average reward of their coalition while interacting non-cooperatively with a finite number of other coalitions. Although the theory has been extensively developed, we are still lacking efficient and scalable computational methods. Here, we develop reinforcement learning methods for such games in a finite space setting with general dynamics and reward functions. We start by proving that MFTG solution yields approximate Nash equilibria in finite-size coalition games. We then propose two algorithms. The first is based on quantization of mean-field spaces and Nash Q-learning. We provide convergence and stability analysis. We then propose a deep reinforcement learning algorithm, which can scale to larger spaces. Numerical experiments in 5 environments with mean-field distributions of dimension up to $200$ show the scalability and efficiency of the proposed method.

Problem

• Develop RL methods for finite space mean-field games
• Prove MFTG solutions yield approximate Nash equilibria
• Scale algorithms to high-dimensional mean-field distributions

Innovation

• Quantization of mean-field spaces with Nash Q-learning
• Deep reinforcement learning for larger spaces
• Scalable algorithms for finite space MFTGs