Simultaneous AlphaZero: Extending Tree Search to Markov Games

2025-12-13

AI Summary

This work addresses two-player, zero-sum, deterministic Markov games with synchronous actions. We propose the first general method extending the AlphaZero framework to this class of games. Our core innovation introduces a bandit-feedback-based regret-minimization matrix game solver at each Monte Carlo Tree Search (MCTS) node, jointly optimizing immediate rewards and value-network predictions to efficiently compute Nash-equilibrium joint actions. Crucially, we achieve end-to-end co-training of the value network, policy network, and online game solver—a novel capability not previously realized. We validate our approach on continuous-state, discrete-action pursuit–evasion and satellite-guardian tasks. The learned policies exhibit robust competitiveness against optimal adversaries, significantly improving both policy quality and convergence stability in synchronous adversarial settings.

Technology Category

• Machine Learning: Online Learning & Bandits
• Game Theory and Economic Paradigms: Adversarial Learning
• Multiagent Systems: Adversarial Agents

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Simultaneous AlphaZero extends the AlphaZero framework to multistep, two-player zero-sum deterministic Markov games with simultaneous actions. At each decision point, joint action selection is resolved via matrix games whose payoffs incorporate both immediate rewards and future value estimates. To handle uncertainty arising from bandit feedback during Monte Carlo Tree Search (MCTS), Simultaneous AlphaZero incorporates a regret-optimal solver for matrix games with bandit feedback. Simultaneous AlphaZero demonstrates robust strategies in a continuous-state discrete-action pursuit-evasion game and satellite custody maintenance scenarios, even when evaluated against maximally exploitative opponents.

Problem

• Extends AlphaZero to simultaneous-action Markov games
• Resolves joint actions via matrix games with rewards
• Handles bandit feedback uncertainty with regret-optimal solvers

Innovation

• Extends AlphaZero to simultaneous-action Markov games
• Resolves joint actions via matrix games with future estimates
• Uses regret-optimal solver for bandit feedback in MCTS