Provably Convergent Actor-Critic in Risk-averse MARL

2026-02-12

Technology Category

• Multiagent Systems: Multiagent Learning
• Game Theory and Economic Paradigms: Adversarial Learning
• Machine Learning: Reinforcement Learning

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Learning stationary policies in infinite-horizon general-sum Markov games (MGs) remains a fundamental open problem in Multi-Agent Reinforcement Learning (MARL). While stationary strategies are preferred for their practicality, computing stationary forms of classic game-theoretic equilibria is computationally intractable -- a stark contrast to the comparative ease of solving single-agent RL or zero-sum games. To bridge this gap, we study Risk-averse Quantal response Equilibria (RQE), a solution concept rooted in behavioral game theory that incorporates risk aversion and bounded rationality. We demonstrate that RQE possesses strong regularity conditions that make it uniquely amenable to learning in MGs. We propose a novel two-timescale Actor-Critic algorithm characterized by a fast-timescale actor and a slow-timescale critic. Leveraging the regularity of RQE, we prove that this approach achieves global convergence with finite-sample guarantees. We empirically validate our algorithm in several environments to demonstrate superior convergence properties compared to risk-neutral baselines.

Problem

• Multi-Agent Reinforcement Learning
• Markov Games
• Stationary Policies
• General-sum Games
• Equilibrium Learning

Innovation

• Risk-averse Quantal Response Equilibrium
• Multi-Agent Reinforcement Learning
• Actor-Critic Algorithm
• Global Convergence
• Two-timescale Learning