Problem
Research questions and friction points this paper is trying to address.
Decentralized MARL
Coarse Correlated Equilibrium
Aggregative Markov Games
Multi-agent Reinforcement Learning
Decentralized MARL for Coarse Correlated Equilibrium in Aggregative Markov Games
🤖 AI Summary
This work addresses the challenge of learning coarse correlated equilibria (CCE) in fully decentralized settings for aggregative Markov games. The authors propose an adaptive stage-based V-learning algorithm that leverages the aggregative structure to design a decentralized CCE learning mechanism for the first time. By employing a two-timescale approach to dynamically adjust stage lengths and integrating no-regret online update strategies, the method achieves model-free scalability to large populations while circumventing the curse of dimensionality inherent in multi-agent systems. Theoretical analysis shows that the algorithm converges to an $\varepsilon$-approximate CCE within $O(S A_{\max} T^5 / \varepsilon^2)$ episodes. Empirical results corroborate both the effectiveness and scalability of the proposed approach.
📝 Abstract
This paper studies the problem of decentralized learning of Coarse Correlated Equilibrium (CCE) in aggregative Markov games (AMGs), where each agent's instantaneous reward depends only on its own action and an aggregate quantity. Existing CCE learning algorithms for general Markov games are not designed to leverage the aggregative structure, and research on decentralized CCE learning for AMGs remains limited. We propose an adaptive stage-based V-learning algorithm that exploits the aggregative structure under a fully decentralized information setting. Based on the two-timescale idea, the algorithm partitions learning into stages and adjusts stage lengths based on the variability of aggregate signals, while using no-regret updates within each stage. We prove the algorithm achieves an epsilon-approximate CCE in O(S Amax T5 / epsilon2) episodes, avoiding the curse of multiagents which commonly arises in MARL. Numerical results verify the theoretical findings, and the decentralized, model-free design enables easy extension to large-scale multi-agent scenarios.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Decentralized MARL
Aggregative Markov Games
Coarse Correlated Equilibrium
Stage-based V-learning
Two-timescale Learning
🔎 Similar Papers
No similar papers found.
