Data-Augmented Game Starts for Accelerating Self-Play Exploration in Imperfect Information Games

2026-05-14

AI Summary

This work addresses the challenge of approximating equilibria in large-scale imperfect-information games, where sparse rewards and long-horizon exploration hinder convergence. To accelerate online exploration in policy gradient methods, the authors propose Data-Augmented Game Starts (DAGS), which leverages intermediate states from offline human expert data as initialization points for self-play—a first in this domain. To mitigate the equilibrium bias introduced by such non-uniform start-state distributions, they introduce a multi-task observation flag mechanism and develop more challenging, analytically tractable benchmark environments. Experimental results demonstrate that, under a fixed computational budget, DAGS substantially reduces exploitability in games such as Kuhn Poker and Goofspiel, significantly improving exploration efficiency.

Abstract

Finding approximate equilibria for large-scale imperfect-information competitive games such as StarCraft, Dota, and CounterStrike remains computationally infeasible due to sparse rewards and challenging exploration over long horizons. In this paper, we propose a multi-agent starting-state sampling strategy designed to substantially accelerate online exploration in regularized policy-gradient game methods for two-player zero-sum (2p0s) games. Motivated by an assumption that offline demonstrations from skilled humans can provide good coverage of high-level strategies relevant to equilibrium play, we propose the initialization of reinforcement learning data collection at intermediate states sampled from offline data to facilitate exploration of strategically relevant subgames. Referring to this method as Data-Augmented Game Starts (DAGS), we perform experiments using synthetic datasets and analytically tractable, long-horizon control variants of two-player Kuhn Poker, Goofspiel, and a counterexample game designed to penalize biased beliefs over hidden information. Under fixed computational budgets, DAGS enables regularized policy gradient methods to achieve lower exploitability in games with significantly more challenging exploration. We show that augmenting starting state distributions when solving imperfect information games can lead to biased equilibria, and we provide a straightforward mitigation to this in the form of multi-task observation flags. Finally, we release a new set of benchmark environments that drastically increase exploration challenges and state counts in existing OpenSpiel games while keeping exploitability measurements analytically tractable.

Problem

• imperfect information games
• approximate equilibria
• exploration
• sparse rewards
• long-horizon

Innovation

• Data-Augmented Game Starts
• imperfect information games
• self-play exploration
• regularized policy gradient
• exploitability