EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games

2026-06-22

AI Summary

This work addresses the inefficiency of exploration in existing policy gradient self-play methods for large-scale two-player zero-sum imperfect-information games, which rely on uniform distribution regularization and struggle to distinguish between superior and inferior actions. To overcome this limitation, the authors propose EMAgnet, a novel approach that employs an exponential moving average (EMA) of the policy network parameters as an adaptive regularization target. This dynamic prior effectively tracks the evolving policy during training, replacing the static uniform prior. Integrated within a PPO-based self-play framework and enhanced with linear and power-law annealing schedules, EMAgnet significantly reduces exploitability across multiple benchmark and high-exploration-difficulty game environments. Notably, it demonstrates superior stability and performance in settings containing a large number of strictly dominated strategies.

Abstract

Recent work has established that regularized policy gradient methods such as PPO, when used in self-play, can match or exceed specialized game-theoretic algorithms for solving two-player zero-sum imperfect-information games. The uniform distribution has emerged as a strong policy regularization target for this purpose, but it regularizes equally toward all actions regardless of their viability. We introduce EMAgnet, which instead regularizes toward an exponential moving average (EMA) of the last-iterate policy's parameters, providing an adaptive regularization target that evolves with the agent's improving strategy. We evaluate EMAgnet on both standard two-player zero-sum benchmarks and modified benchmarks with exploration challenges and large numbers of strictly dominated strategies. Relative to PPO self-play with uniform-magnet regularization under both linear and power-law annealing schedules, EMAgnet achieves lower exploitability in the majority of tested environments, with consistent performance gains across games containing strictly dominated strategies.

Problem

• policy gradient
• self-play
• regularization
• imperfect-information games
• exploitability

Innovation

• EMA regularization
• policy gradient self-play
• adaptive regularization
• imperfect-information games
• exploitability