Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs

2026-06-13

AI Summary

Existing approaches struggle to scalably and reproducibly evaluate large language models’ step-by-step reasoning capabilities in complex mathematical proofs. This work proposes an automated pipeline that transforms real mathematical proofs into masked-step reconstruction tasks, integrating context extraction and an LLM-driven multi-round voting mechanism for equivalence judgment to enable fine-grained reasoning assessment. The authors introduce Mask-ProofBench, a high-quality benchmark comprising 292 cross-domain problems, and validate its effectiveness across 17 models. Models enhanced for reasoning exhibit performance gains of 12%–27%, and evaluation results achieve 96.8% agreement with expert annotations, substantially improving the reliability and comparability of mathematical reasoning evaluation.

Abstract

Large language models (LLMs) are increasingly capable of mathematical problem solving and can even assist with research-level proofs, yet we still lack a scalable and reproducible way to measure step-level reasoning in long proofs across diverse sources. This evaluation gap limits trustworthy AI assistance in proof-certified scientific progress. Existing evaluations often emphasize final answers or rely on costly expert grading, while end-to-end proof generation remains open-ended and hard to verify automatically. We introduce Mask-Proof, a pipeline that turns real proofs into automatically checkable masked-step tasks. It masks key formula steps, provides the necessary surrounding context, and evaluates model reconstructions with an LLM-based equivalence judge using repeated votes for stability. The resulting Mask-ProofBench contains 292 curated problems across diverse research areas. Experiments with 17 models show that reasoning-enhanced models outperform standard models by 12% to 27%. Our evaluator achieves 96.8% agreement with expert annotators, enabling faithful, reproducible, and comparable measurement of step-level mathematical reasoning. Benchmark, annotations, and code are available at https://github.com/weating/Mask-Proof.

Problem

• mathematical reasoning
• step-level evaluation
• automated proof verification
• large language models
• data curation

Innovation

• masked-step evaluation
• mathematical reasoning
• LLM-based equivalence judging
• automated proof curation
• step-level reasoning benchmark