Rethinking Math Reasoning Evaluation: A Robust LLM-as-a-Judge Framework Beyond Symbolic Rigidity

2026-04-24

AI Summary

This work addresses the limitations of current mathematical reasoning evaluation methods, which rely on symbolic matching and struggle to accommodate diverse answer formats and expressions. To overcome this, the authors propose a large language model (LLM)-based framework for assessing semantic equivalence, replacing traditional rule-driven symbolic comparison. By leveraging the LLM’s deep semantic understanding, the approach transcends the rigidity of exact symbolic matching, substantially enhancing the robustness and generalizability of evaluation. Experimental results on established benchmarks such as Lighteval and SimpleRL demonstrate that the proposed framework significantly reduces misjudgments compared to conventional methods, thereby improving both the accuracy and reliability of mathematical answer assessment.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Evaluation and Analysis
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent advancements in large language models have led to significant improvements across various tasks, including mathematical reasoning, which is used to assess models' intelligence in logical reasoning and problem-solving. Models are evaluated on mathematical reasoning benchmarks by verifying the correctness of the final answer against a ground truth answer. A common approach for this verification is based on symbolic mathematics comparison, which fails to generalize across diverse mathematical representations and solution formats. In this work, we offer a robust and flexible alternative to rule-based symbolic mathematics comparison. We propose an LLM-based evaluation framework for evaluating model-generated answers, enabling accurate evaluation across diverse mathematical representations and answer formats. We present failure cases of symbolic evaluation in two popular frameworks, Lighteval and SimpleRL, and compare them to our approach, demonstrating clear improvements over commonly used methods. Our framework enables more reliable evaluation and benchmarking, leading to more accurate performance monitoring, which is important for advancing mathematical problem-solving and intelligent systems.

Problem

• mathematical reasoning
• LLM evaluation
• symbolic comparison
• answer verification
• benchmarking

Innovation

• LLM-as-a-Judge
• mathematical reasoning evaluation
• symbolic comparison
• answer format robustness
• flexible evaluation framework