🤖 AI Summary
This work addresses the challenge of modeling belief states in partially observable, stochastic multi-agent games, where existing approaches such as MuZero struggle to effectively represent hidden information. The authors propose a belief-aware mechanism that operates within the MuZero framework without requiring explicit belief tracking or modifications to the search algorithm. By introducing a self-conditioned auxiliary prediction head trained via supervisory signals from winner prediction and rank estimation, the latent state representation is encouraged to retain outcome-predictive information. Integrated with a Transformer encoder, Monte Carlo tree search, heuristic opponent curriculum learning, and self-play training, the method achieves a 75.3% win rate (equivalent to +194 Elo) against baseline agents in the card game Skyjo, and improves the win rate against heuristic opponents from 0.466 to 0.720, substantially outperforming prior approaches.
📝 Abstract
In 2019, Google DeepMind released MuZero, a model-based reinforcement learning method that achieves strong results in perfect-information games by combining learned dynamics models with Monte Carlo Tree Search (MCTS). However, comparatively little work has extended MuZero to partially observable, stochastic, multi-player environments, where agents must act under uncertainty about hidden state. Such settings arise not only in card games but in domains such as autonomous negotiation, financial trading, and multi-agent robotics. In the absence of explicit belief modeling, MuZero's latent encoding has no dedicated mechanism for representing uncertainty over unobserved variables.
To address this, we introduce SkyNet (Belief-Aware MuZero), which adds ego-conditioned auxiliary heads for winner prediction and rank estimation to the standard MuZero architecture. These objectives encourage the latent state to retain information predictive of outcomes under partial observability, without requiring explicit belief-state tracking or changes to the search algorithm.
We evaluate SkyNet on Skyjo, a partially observable, non-zero-sum, stochastic card game, using a decision-granularity environment, transformer-based encoding, and a curriculum of heuristic opponents with self-play. In 1000-game head-to-head evaluations at matched checkpoints, SkyNet achieves a 75.3% peak win rate against the baseline (+194 Elo, $p < 10^{-50}$). SkyNet also outperforms the baseline against heuristic opponents (0.720 vs.\ 0.466 win rate). Critically, the belief-aware model initially underperforms the baseline but decisively surpasses it once training throughput is sufficient, suggesting that belief-aware auxiliary supervision improves learned representations under partial observability, but only given adequate data flow.