On-line Policy Improvement using Monte-Carlo Search
This paper addresses the low decision quality and high error rates of controllers in real-time adaptive control. We propose an online Monte Carlo Policy Improvement (MCPI) algorithm that requires neither an environmental model nor gradient information, relying solely on a simulatable environment and an initial policy. MCPI estimates long-term action returns via parallel multi-step random rollouts and dynamically updates the policy. Its key innovation lies in directly applying a lightweight, scalable Monte Carlo Tree Search (MCTS) for online policy optimization, enabling plug-and-play reinforcement learning enhancement. Evaluated on backgammon, MCPI reduces decision error rates by over fivefold compared to baselines—including random policies and TD-Gammon—demonstrating strong generalization capability and real-time efficacy in practical adaptive control scenarios.