Process-Verified Reinforcement Learning for Theorem Proving via Lean

2026-06-18

AI Summary

This work addresses the limitation of traditional reinforcement learning in theorem proving, which relies solely on binary verification signals and fails to exploit fine-grained feedback from the proof process. We propose the first approach that leverages the Lean proof assistant as a process-level reward oracle during training, utilizing its elaboration mechanism to parse tactic sequences, identify locally valid steps, and pinpoint the first failure point, thereby generating type-theoretically grounded dense credit signals. Our method introduces a reinforcement learning objective that combines both outcome- and process-based advantages, along with novel mechanisms for first-failure propagation and first-token credit assignment. Experiments on STP-Lean and DeepSeek-Prover-V1.5 demonstrate substantial improvements over result-only feedback baselines on benchmarks including MiniF2F and ProofNet.

Abstract

While reinforcement learning from verifiable rewards (RLVR) typically has relied on a single binary verification signal, symbolic proof assistants in formal reasoning offer rich, fine-grained structured feedback. This gap between structured processes and unstructured rewards highlights the importance of feedback that is both dense and sound. In this work, we demonstrate that the Lean proof assistant itself can serve as a symbolic process oracle, supplying both outcome-level and fine-grained tactic-level verified feedback during training. Proof attempts are parsed into tactic sequences, and Lean's elaboration marks both locally sound steps and the earliest failing step, yielding dense, verifier-grounded credit signals rooted in type theory. We incorporate these structured rewards into a GRPO-style reinforcement learning objective with first-error propagation and first-token credit methods that balances outcome- and process-level advantages. Experiments with STP-Lean and DeepSeek-Prover-V1.5 show that tactic-level supervision outperforms outcome-only baselines in most settings, delivering improvements on benchmarks such as MiniF2F and ProofNet. Beyond empirical gains, our study highlights a broader perspective: symbolic proof assistants are not only verifiers at evaluation time, but can also act as process-level reward oracles during training. This opens a path toward reinforcement learning frameworks that combine the scalability of language models with the reliability of symbolic verification for formal reasoning.

Problem

• reinforcement learning
• theorem proving
• formal verification
• process feedback
• proof assistants

Innovation

• process-verified reinforcement learning
• Lean proof assistant
• tactic-level feedback
• symbolic process oracle
• dense verified rewards