🤖 AI Summary
This work identifies the fundamental cause of generalization failure in AlphaZero-style reinforcement learning on impartial games (e.g., Nim): neural networks relying on local observations cannot implicitly learn global, non-local abstract functions—such as parity—that determine game-theoretic outcomes, and state-outcome correlations vanish identically. We construct an AlphaZero variant integrating self-play training, Monte Carlo Tree Search, and residual CNNs, augmented with state-masking analysis and systematic generalization diagnostics. For the first time, we rigorously establish that the decoupling between local observability and global win-loss determination constitutes the core bottleneck for RL success in such domains, while also linking this failure to data skew and label noise. Experiments show convergence on small Nim instances, but training efficiency collapses catastrophically with scale; moreover, value networks fail to infer outcomes from partial states, exhibiting markedly lower robustness than in biased games like Chess or Go.
📝 Abstract
While AlphaZero-style reinforcement learning (RL) algorithms excel in various board games, in this paper we show that they face challenges on impartial games where players share pieces. We present a concrete example of a game - namely the children's game of Nim - and other impartial games that seem to be a stumbling block for AlphaZero-style and similar self-play reinforcement learning algorithms. Our work is built on the challenges posed by the intricacies of data distribution on the ability of neural networks to learn parity functions, exacerbated by the noisy labels issue. Our findings are consistent with recent studies showing that AlphaZero-style algorithms are vulnerable to adversarial attacks and adversarial perturbations, showing the difficulty of learning to master the games in all legal states. We show that Nim can be learned on small boards, but the learning progress of AlphaZero-style algorithms dramatically slows down when the board size increases. Intuitively, the difference between impartial games like Nim and partisan games like Chess and Go can be explained by the fact that if a small part of the board is covered for impartial games it is typically not possible to predict whether the position is won or lost as there is often zero correlation between the visible part of a partly blanked-out position and its correct evaluation. This situation starkly contrasts partisan games where a partly blanked-out board position typically provides abundant or at least non-trifle information about the value of the fully uncovered position.
