Video Game Level Design as a Multi-Agent Reinforcement Learning Problem

2025-10-06

AI Summary

Existing PCGRL approaches rely on single-agent architectures, encountering two key bottlenecks in large-scale level generation: (1) computationally expensive global heuristic reward computation, leading to poor efficiency; and (2) weak generalization to out-of-distribution level geometries. To address these, we propose Multi-Agent PCGRL (MA-PCGRL), a collaborative generative framework that models level construction as a distributed, local decision-making task. Each agent observes and modifies only its local region, while sharing lightweight, decentralized reward signals—eliminating redundant global metric evaluations. Spatial decoupling and parameter sharing are achieved via modular policy learning. Experiments demonstrate that MA-PCGRL maintains competitive generation quality while significantly reducing computational overhead, improving generalization to unseen map shapes, and enhancing training stability. Our approach establishes a new paradigm for scalable and robust procedural content generation.

Technology Category

• Multiagent Systems: Multiagent Learning
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Procedural Content Generation via Reinforcement Learning (PCGRL) offers a method for training controllable level designer agents without the need for human datasets, using metrics that serve as proxies for level quality as rewards. Existing PCGRL research focuses on single generator agents, but are bottlenecked by the need to frequently recalculate heuristics of level quality and the agent's need to navigate around potentially large maps. By framing level generation as a multi-agent problem, we mitigate the efficiency bottleneck of single-agent PCGRL by reducing the number of reward calculations relative to the number of agent actions. We also find that multi-agent level generators are better able to generalize to out-of-distribution map shapes, which we argue is due to the generators' learning more local, modular design policies. We conclude that treating content generation as a distributed, multi-agent task is beneficial for generating functional artifacts at scale.

Problem

• Addresses efficiency bottlenecks in single-agent procedural content generation
• Enhances generalization to out-of-distribution map shapes via multi-agent learning
• Develops modular design policies for scalable functional artifact generation

Innovation

• Multi-agent reinforcement learning for level design
• Reduces reward calculations to improve efficiency
• Learns local modular policies for generalization