What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning

2024-12-20

arXiv.org

AI Summary

The underlying mechanisms driving the effectiveness of stepwise reward models (SRMs) in mathematical reasoning—particularly their differential sensitivity to natural language versus formal mathematical language—remain poorly understood. Method: We employ Monte Carlo Tree Search (MCTS) to automatically generate preference annotations, coupled with stepwise preference modeling, reinforcement learning supervision, and rigorous analysis of reasoning trajectories. Contribution/Results: We find that SRMs primarily evaluate the logical coherence and formal structure of mathematical reasoning chains—not natural language descriptions. Empirically, removing natural language components from chain-of-thought (CoT) reasoning degrades performance negligibly, while SRMs exhibit high sensitivity to formal logical structure yet surprising robustness to semantic variations in natural language. This reveals SRMs as *logical structure evaluators*, fundamentally distinct from linguistic or semantic reward models. Our findings establish a theoretical foundation and a novel paradigm for lightweight, interpretable, and computationally efficient reward modeling in mathematical reasoning.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Step-level reward models (SRMs) can significantly enhance mathematical reasoning performance through process supervision or step-level preference alignment based on reinforcement learning. The performance of SRMs is pivotal, as they serve as critical guidelines, ensuring that each step in the reasoning process is aligned with desired outcomes. Recently, AlphaZero-like methods, where Monte Carlo Tree Search (MCTS) is employed for automatic step-level preference annotation, have proven particularly effective. However, the precise mechanisms behind the success of SRMs remain largely unexplored. To address this gap, this study delves into the counterintuitive aspects of SRMs, particularly focusing on MCTS-based approaches. Our findings reveal that the removal of natural language descriptions of thought processes has minimal impact on the efficacy of SRMs. Furthermore, we demonstrate that SRMs are adept at assessing the complex logical coherence present in mathematical language while having difficulty in natural language. These insights provide a nuanced understanding of the core elements that drive effective step-level reward modeling in mathematical reasoning. By shedding light on these mechanisms, this study offers valuable guidance for developing more efficient and streamlined SRMs, which can be achieved by focusing on the crucial parts of mathematical reasoning.

Problem

• Hierarchical Reward Models
• Mathematical Reasoning
• Natural Language vs Mathematical Language

Innovation

• Hierarchical Reward Models
• Mathematical Reasoning
• Reinforcement Learning and Search Techniques