RPM-MCTS: Knowledge-Retrieval as Process Reward Model with Monte Carlo Tree Search for Code Generation

2025-11-24

AI Summary

Existing tree-search-based code generation methods suffer from ineffective intermediate-step evaluation and poor error localization, leading to erroneous outputs and high computational overhead. This paper proposes RPM-MCTS, a novel approach that replaces conventional *trained* process reward models with *knowledge retrieval* to enable fine-grained assessment of algorithmic reasoning paths. It further enhances exploration diversity via similarity-based filtering and incorporates sandbox execution feedback for precise error localization and targeted correction. Evaluated on four public benchmarks, RPM-MCTS significantly outperforms prior methods while reducing token consumption by approximately 15%. Additionally, full fine-tuning of the base model on high-quality synthetic code data systematically improves its code generation capability. The core innovations are: (i) a training-free process reward modeling mechanism grounded in knowledge retrieval, and (ii) an execution-driven, iterative error correction framework.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Tree search-based methods have made significant progress in enhancing the code generation capabilities of large language models. However, due to the difficulty in effectively evaluating intermediate algorithmic steps and the inability to locate and timely correct erroneous steps, these methods often generate incorrect code and incur increased computational costs. To tackle these problems, we propose RPM-MCTS, an effective method that utilizes Knowledge-Retrieval as Process Reward Model based on Monte Carlo Tree Search to evaluate intermediate algorithmic steps. By utilizing knowledge base retrieval, RPM-MCTS avoids the complex training of process reward models. During the expansion phase, similarity filtering is employed to remove redundant nodes, ensuring diversity in reasoning paths. Furthermore, our method utilizes sandbox execution feedback to locate erroneous algorithmic steps during generation, enabling timely and targeted corrections. Extensive experiments on four public code generation benchmarks demonstrate that RPM-MCTS outperforms current state-of-the-art methods while achieving an approximately 15% reduction in token consumption. Furthermore, full fine-tuning of the base model using the data constructed by RPM-MCTS significantly enhances its code capabilities.

Problem

• Evaluating intermediate algorithmic steps in code generation is challenging
• Existing methods cannot locate and correct erroneous steps timely
• Tree search approaches generate incorrect code with high computational costs

Innovation

• Knowledge-Retrieval Process Reward Model avoids complex training
• Similarity filtering removes redundant nodes during expansion
• Sandbox execution feedback locates and corrects erroneous steps