A Comprehensive Review of Multi-Agent Reinforcement Learning in Video Games

2025-09-03

AI Summary

This paper systematically reviews advances and challenges in applying multi-agent reinforcement learning (MARL) to video games. Addressing core issues—including non-stationarity, partial observability, sparse rewards, collaborative modeling, and scalability—across turn-based two-player games to real-time multiplayer online battle arenas (MOBAs), real-time strategy (RTS), and first-person shooters (FPS), we propose a task-dimensional and interaction-complexity-based game difficulty assessment framework. We unify the analysis of MARL methodologies across representative games such as *StarCraft II*, *Dota 2*, and *Honor of Kings*, covering self-play, centralized training with decentralized execution (CTDE), hierarchical cooperation, and curriculum learning. For the first time, we conduct cross-game challenge categorization and performance attribution analysis, identifying empirically verifiable future research directions. Our work bridges the gap between algorithmic validation and robust deployment of MARL in complex, open-ended environments.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Reinforcement Learning
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation

Abstract

Recent advancements in multi-agent reinforcement learning (MARL) have demonstrated its application potential in modern games. Beginning with foundational work and progressing to landmark achievements such as AlphaStar in StarCraft II and OpenAI Five in Dota 2, MARL has proven capable of achieving superhuman performance across diverse game environments through techniques like self-play, supervised learning, and deep reinforcement learning. With its growing impact, a comprehensive review has become increasingly important in this field. This paper aims to provide a thorough examination of MARL's application from turn-based two-agent games to real-time multi-agent video games including popular genres such as Sports games, First-Person Shooter (FPS) games, Real-Time Strategy (RTS) games and Multiplayer Online Battle Arena (MOBA) games. We further analyze critical challenges posed by MARL in video games, including nonstationary, partial observability, sparse rewards, team coordination, and scalability, and highlight successful implementations in games like Rocket League, Minecraft, Quake III Arena, StarCraft II, Dota 2, Honor of Kings, etc. This paper offers insights into MARL in video game AI systems, proposes a novel method to estimate game complexity, and suggests future research directions to advance MARL and its applications in game development, inspiring further innovation in this rapidly evolving field.

Problem

• Reviewing MARL applications in diverse video game genres
• Analyzing challenges like nonstationarity and partial observability in games
• Proposing methods to estimate game complexity for MARL

Innovation

• Multi-agent reinforcement learning for diverse games
• Self-play and deep reinforcement learning techniques
• Novel method to estimate game complexity