Problem
Research questions and friction points this paper is trying to address.
data contamination
large language models
mathematical problem solving
benchmark rewriting
answer correctness
RePro: Proof-Verified Benchmark Rewriting for Reliable Evaluation of LLM Mathematical Problem Solving
🤖 AI Summary
为解决数据污染影响大模型数学解题评估的问题，提出RePro框架，利用神经自动定理证明器重写问题并确保答案正确性。
📝 Abstract
Data contamination undermines the reliable evaluation of large language models (LLMs) on mathematical problem solving. While rewriting-based evaluation mitigates memorization, existing methods lack guarantees of problem validity and answer correctness. We propose Proof-Verified Benchmark Rewriting (RePro), the first framework to integrate Lean-oriented neural automated theorem provers (ATPs) into benchmark rewriting, which rewrites problems and regenerates answers with correctness ensured by Lean-verified proofs. Experiments on GSM8K and MATH show that RePro's retained rewritten instances achieve 100% well-definedness, feasibility, and answer correctness, while existing methods still produce invalid or incorrect instances. Moreover, several models exhibit accuracy drops on proof-verified rewritten benchmarks, suggesting that their performance is sensitive to surface-level and structural variations and may partly reflect memorization effects. Our source code and data are available at https://github.com/AI4Engi/RePro.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Proof-Verified Benchmark Rewriting
Neural Automated Theorem Provers
Benchmark Rewriting
Answer Correctness
Memorization Effects
🔎 Similar Papers
💼 Related Jobs
