Evaluating Mathematical Reasoning Beyond Accuracy

2024-04-08

arXiv.org

AI Summary

Existing mathematical reasoning evaluations solely assess final answer correctness, neglecting the quality of intermediate reasoning steps—thus failing to detect logical fallacies or redundant derivations. To address this, we propose ReasonEval, the first systematic framework for evaluating the quality of intermediate steps in large language model (LLM)-generated mathematical reasoning. ReasonEval innovatively quantifies reasoning quality along two orthogonal dimensions: **effectiveness** (logical correctness of each step) and **redundancy** (necessity of each step). Our method leverages a mathematically knowledgeable base model and fine-tunes an LLM-based evaluator on high-quality human-annotated data, enabling scalable, automated, step-level assessment. Experiments demonstrate that high answer accuracy does not imply high reasoning quality; ReasonEval significantly outperforms baseline evaluators across multiple meta-evaluation datasets. We open-source our best-performing model, evaluation scripts, and full results—enabling both data filtering and robust, generalizable meta-evaluation.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Machine Learning: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The leaderboard of Large Language Models (LLMs) in mathematical tasks has been continuously updated. However, the majority of evaluations focus solely on the final results, neglecting the quality of the intermediate steps. This oversight can mask underlying problems, such as logical errors or unnecessary steps in the reasoning process. To measure reasoning beyond final-answer accuracy, we introduce ReasonEval, a new methodology for evaluating the quality of reasoning steps. ReasonEval employs validity and redundancy to characterize the reasoning quality, as well as accompanying LLMs to assess them automatically. We explore different design options for the LLM-based evaluators and empirically demonstrate that ReasonEval, when instantiated with base models possessing strong mathematical knowledge and trained with high-quality labeled data, consistently outperforms baseline methods in the meta-evaluation datasets. We also highlight the strong generalization capabilities of ReasonEval. By utilizing ReasonEval to evaluate LLMs specialized in math, we find that an increase in final-answer accuracy does not necessarily guarantee an improvement in the overall quality of the reasoning steps for challenging mathematical problems. Additionally, we observe that ReasonEval can play a significant role in data selection. We open-source the best-performing model, meta-evaluation script, and all evaluation results to facilitate future research.

Problem

• Large Language Models
• Mathematical Problem Solving
• Reasoning Evaluation

Innovation

• ReasonEval
• Mathematical Reasoning Assessment
• Large Language Models