FormalRewardBench: A Benchmark for Formal Theorem Proving Reward Models

2026-05-11

AI Summary

This work addresses the challenges of sparse reward signals and the difficulty of evaluating partially correct proofs in reinforcement learning for formal theorem proving. It introduces the first benchmark for proof reward models tailored to Lean 4, constructed from 250 expert-curated preference pairs of correct and incorrect proofs. The benchmark incorporates five fine-grained error injection strategies to systematically assess a model’s ability to discriminate valid from invalid proofs. Experimental results reveal that state-of-the-art large language models achieve the highest accuracy (59.8%), while specialized theorem provers perform worst (24.4%), highlighting significant limitations in current models’ capacity for proof evaluation. This benchmark thus offers a novel and valuable tool for measuring and advancing proof assessment capabilities in automated reasoning systems.

Abstract

Recent neural theorem provers use reinforcement learning with verifiable rewards (RLVR), where proof assistants provide binary correctness signals. While verifiable rewards are cheap and scalable without reward hacking issues, they suffer from sparse credit assignment: models receive no learning signal from difficult problems where partial progress goes unrewarded. This motivates learned reward models that can evaluate proof quality beyond binary verification. However, comparing reward models is challenging since it typically requires expensive RL training ablations. To address this, we introduce \textbf{FormalRewardBench}, the first benchmark for evaluating reward models in formal theorem proving with Lean 4. Our benchmark consists of 250 preference pairs where correct proofs are paired with incorrect variants generated through five expert curated error injection strategies: forced mistakes, minimal single-point variations, verbose incorrect proofs, natural language justification, and Python code injection. We evaluate frontier LLMs (e.g., Claude Opus 4.5), judge LLMs (e.g., CompassJudger-1-14B), general-purpose LLMs (e.g., Qwen2.5-72B-Instruct), and specialized theorem proving models (e.g., DeepSeek-Prover-V2-7B). Our results reveal that frontier LLMs achieve the highest performance (59.8\%) while specialized theorem provers perform the worst (24.4\%), suggesting that theorem proving ability does not transfer to proof evaluation. We provide further insights on various error injection mechanisms, highlighting the challenging nature of most injection mechanisms. We release \textbf{FormalRewardBench} publicly to encourage more research on developing reward models in formal mathematics.

Problem

• formal theorem proving
• reward models
• credit assignment
• proof evaluation
• benchmark

Innovation

• FormalRewardBench
• reward modeling
• theorem proving
• preference benchmark
• error injection