Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations

2026-08-26

AI Summary

本文探讨了在大规模多智能体系统中，通过学习低维群体表示来解决高维控制问题，从而优化策略并提高奖励预测和纳什均衡质量。

Abstract

Modern multi-agent systems are increasingly deployed at scale over large populations of agents in settings such as ad-auctions, traffic routing, and recommendation systems. The dominant approach in such settings is to optimize each agent's policy independently, treating the other agents as part of a fixed single-agent environment rather than modeling the population dynamics. In many large-population systems, the dynamics depend on an aggregate summary of the population rather than the identity of any individual. Mean-field RL exploits such structure, providing a principled framework that models each agent's environment as an explicit function of the population distribution. However, in large state-action spaces or high-dimensional control problems, modeling the population distribution is itself intractable. How can we design a scalable framework for high-dimensional control problems with large populations? This work explores this question from the perspective of representation learning. We introduce a mean-field RL framework in which the rewards and transition dynamics depend on the population only through an unknown low-dimensional aggregate statistic. We then study this framework in the offline setting and design a provable approach that learns a near-optimal policy by learning a low-dimensional representation. Motivated by real-life supply-chain optimization problems, we design a one-step routing game to test the hypothesis that learning a low-dimensional population representation improves reward prediction and Nash gap estimation relative to baselines that don't exploit this structure. We show that under a fixed neural-network parameter count and optimization budget, learning a low-dimensional population representation improves reward prediction and the equilibrium quality of the resulting policies.

Problem

• Reinforcement Learning
• Large Populations
• High-Dimensional Control Problems
• Representation Learning
• Mean-Field

Innovation

• Mean-field RL
• Representation Learning
• Low-dimensional Aggregate Statistic
• Offline Setting
• Scalable Framework