LeanProgress: Guiding Search for Neural Theorem Proving via Proof Progress Prediction

2025-02-25

AI Summary

Large language models (LLMs) lack awareness of proof progress when conducting long-range formal theorem proving in Lean, resulting in inefficient search and suboptimal user-guided tactic selection. To address this, we propose the first learnable proof progress prediction mechanism, formulated as a regression task that estimates the remaining number of steps to proof completion—thereby enabling dynamic, quantitative assessment of proof state advancement. Our method leverages a large-scale corpus derived from Lean Workbook Plus and Mathlib4, incorporates data rebalancing, and integrates seamlessly with Best-First Search within the Reprover framework. Experimental evaluation demonstrates that our progress predictor achieves 75.1% accuracy in step-count estimation. When deployed in Reprover, it improves overall proof success rate on Mathlib4 from 41.2% to 45.0%, with particularly pronounced gains for long proofs—marking the first effective solution for adaptive progress-aware formal reasoning.

Technology Category

• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Mathematical reasoning remains a significant challenge for Large Language Models (LLMs) due to hallucinations. When combined with formal proof assistants like Lean, these hallucinations can be eliminated through rigorous verification, making theorem proving reliable. However, even with formal verification, LLMs still struggle with long proofs and complex mathematical formalizations. While Lean with LLMs offers valuable assistance with retrieving lemmas, generating tactics, or even complete proofs, it lacks a crucial capability: providing a sense of proof progress. This limitation particularly impacts the overall development efficiency in large formalization projects. We introduce LeanProgress, a method that predicts the progress in the proof. Training and evaluating our models made on a large corpus of Lean proofs from Lean Workbook Plus and Mathlib4 and how many steps remain to complete it, we employ data preprocessing and balancing techniques to handle the skewed distribution of proof lengths. Our experiments show that LeanProgress achieves an overall prediction accuracy of 75.1% in predicting the amount of progress and, hence, the remaining number of steps. When integrated into a best-first search framework using Reprover, our method shows a 3.8% improvement on Mathlib4 compared to baseline performances of 41.2%, particularly for longer proofs. These results demonstrate how proof progress prediction can enhance both automated and interactive theorem proving, enabling users to make more informed decisions about proof strategies.

Problem

• Predicts proof progress in theorem proving
• Enhances efficiency in formal verification
• Improves accuracy in long proof strategies

Innovation

• Predicts proof progress accurately
• Integrates with best-first search
• Enhances theorem proving efficiency