Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness

2026-05-11

AI Summary

Current evaluations of mathematical proofs generated by large language models predominantly focus on correctness, often overlooking critical dimensions such as clarity, conciseness, insightfulness, and transferability. This work introduces ProofRank, a novel benchmark that systematically defines and quantifies five scalable quality metrics: conciseness, computational simplicity, cognitive simplicity, diversity, and adaptability. Leveraging a dataset derived from mathematical competition problems, the study conducts a comprehensive evaluation of mainstream models using automated proxy metrics. The results reveal substantial variation in proof quality across models and demonstrate that the most correct proofs are not necessarily the highest-quality ones, highlighting a trade-off between correctness and holistic proof quality. These findings underscore the need for future evaluation frameworks to balance practical utility with formal correctness.

Abstract

Large language models (LLMs) have become capable mathematical problem-solvers, often producing correct proofs for challenging problems. However, correctness alone is not sufficient: mathematical proofs should also be clear, concise, insightful, and transferable to other problems. While this proof quality is subjective and depends on the reader and context, many of its components are concrete and broadly valued. In this work, we identify such components and introduce ProofRank, a benchmark curated from challenging mathematical competitions. ProofRank evaluates several scalable proxies of proof quality: (i) conciseness, measuring whether proofs avoid unnecessary steps; (ii) computational ease, measuring the extent to which a proof relies on tedious calculations; (iii) cognitive simplicity, measuring how accessible the used proof techniques are; (iv) diversity, measuring how varied a model's proofs for a single problem are; and (v) adaptivity, measuring whether a model can follow a specified proof technique. Across models, we find substantial differences in proof quality that are not captured by correctness-only benchmarks. We also observe significant trade-offs between proof-quality metrics and correctness, suggesting that future evaluations of mathematical reasoning should measure how useful LLM-generated proofs are.

Problem

• proof quality
• large language models
• mathematical reasoning
• correctness
• evaluation benchmark

Innovation

• proof quality
• ProofRank
• mathematical reasoning
• large language models
• evaluation benchmark