PMCTS: Particle Monte Carlo Tree Search for Principled Parallelized Inference Time Scaling

2026-05-09

AI Summary

This work proposes Particle Monte Carlo Tree Search (PMCTS), a novel parallel framework for Monte Carlo Tree Search (MCTS) that overcomes the inherent sequential limitations of traditional MCTS in parallel environments. PMCTS introduces a particle-based sampling mechanism that effectively integrates neural network policies and value functions, enabling large-scale parallel inference while preserving theoretical guarantees for policy improvement—the first such guarantee in parallel MCTS. The method maintains rigorous theoretical foundations and demonstrates substantial performance gains over existing heuristic parallel baselines, achieving superior scalability and consistent improvements across multiple tasks.

Abstract

Monte Carlo Tree Search (MCTS) is a widely used approach for policy improvement through search with increasing popularity for real world applications. Due to the sequential and deterministic nature of its search, runtime-scaling of MCTS with parallel compute remains a major challenge. We introduce Particle MCTS (PMCTS), to our knowledge the first principled parallel MCTS algorithm which is suited for neural network evaluations and can preserve formal policy improvement guarantees. Empirically, PMCTS scales well with parallel compute and significantly outperforms the popular heuristic-based baselines across domains.

Problem

• Monte Carlo Tree Search
• parallelization
• inference time scaling
• policy improvement
• runtime scaling

Innovation

• Particle MCTS
• parallelized inference
• Monte Carlo Tree Search
• policy improvement guarantees
• neural network evaluation