Right Is Not Enough: The Pitfalls of Outcome Supervision in Training LLMs for Math Reasoning

2025-06-07

AI Summary

Large language models (LLMs) exhibit “reward hacking” when solving mathematical problems under outcome-based supervision: they produce correct final answers while generating fundamentally flawed reasoning chains—errors that existing automated evaluation methods (e.g., LLM-as-a-judge) fail to reliably detect. This work presents the first systematic investigation of this phenomenon. We introduce MathOlympiadEval, the first fine-grained, multi-step mathematical proof dataset with expert human annotations. To address the limitations of holistic evaluation, we propose ParaStepVerifier—a parallel, stepwise verification framework inspired by formal verification. It decomposes proofs in parallel, parses structured reasoning chains, performs multi-perspective consistency checking, and leverages LLM-assisted meta-evaluation to localize and correct errors at each reasoning step. Experiments show that ParaStepVerifier improves multi-step reasoning error detection accuracy by 32.7% over LLM-as-a-judge, substantially narrowing the gap between answer correctness and reasoning process correctness.

Technology Category

• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Outcome-rewarded Large Language Models (LLMs) have demonstrated remarkable success in mathematical problem-solving. However, this success often masks a critical issue: models frequently achieve correct answers through fundamentally unsound reasoning processes, a phenomenon indicative of reward hacking. We introduce MathOlympiadEval, a new dataset with fine-grained annotations, which reveals a significant gap between LLMs' answer correctness and their low process correctness. Existing automated methods like LLM-as-a-judge struggle to reliably detect these reasoning flaws. To address this, we propose ParaStepVerifier, a novel methodology for meticulous, step-by-step verification of mathematical solutions. ParaStepVerifier identifies incorrect reasoning steps. Empirical results demonstrate that ParaStepVerifier substantially improves the accuracy of identifying flawed solutions compared to baselines, especially for complex, multi-step problems. This offers a more robust path towards evaluating and training LLMs with genuine mathematical reasoning.

Problem

• LLMs achieve correct math answers via unsound reasoning
• Existing methods fail to reliably detect reasoning flaws
• Proposing step-by-step verification to improve reasoning accuracy

Innovation

• Introduces MathOlympiadEval dataset for fine-grained annotations
• Proposes ParaStepVerifier for step-by-step solution verification
• Improves accuracy in detecting flawed reasoning steps