Online Robust Reinforcement Learning Through Monte-Carlo Planning

2026-09-16

AI Summary

本文提出了一种新的鲁棒MCTS算法，通过引入鲁棒幂平均备份算子和精心设计的探索奖励来解决低保真度模拟器中的动态模型不确定性问题。

Abstract

Monte Carlo Tree Search (MCTS) is a powerful framework for solving complex decision-making problems, yet it often relies on the assumption that the simulator and the real-world dynamics are identical. Although this assumption helps achieve the success of MCTS in games like Chess, Go, and Shogi, the real-world scenarios incur ambiguity due to their modeling mismatches in low-fidelity simulators. In this work, we present a new robust variant of MCTS that mitigates dynamical model ambiguities. Our algorithm addresses transition dynamics and reward distribution ambiguities to bridge the gap between simulation-based planning and real-world deployment. We incorporate a robust power mean backup operator and carefully designed exploration bonuses to ensure finite-sample convergence at every node in the search tree. We show that our algorithm achieves a convergence rate of $\mathcal{O}(n^{-1/2})$ for the value estimation at the root node, comparable to that of standard MCTS. Finally, we provide empirical evidence that our method achieves robust performance in planning problems even under significant ambiguity in the underlying reward distribution and transition dynamics.

Problem

• Monte Carlo Tree Search
• dynamical model ambiguities
• real-world deployment
• transition dynamics
• reward distribution

Innovation

• Robust MCTS
• Power Mean Backup Operator
• Exploration Bonuses
• Model Ambiguity
• Finite-Sample Convergence