Reinforced Large Language Model is a formal theorem prover

2025-02-13

AI Summary

This work addresses the poor generalization and severe exposure bias of large language models (LLMs) in formal theorem proving. Methodologically, it proposes a fine-tuning-free reinforcement learning framework wherein the LLM serves as a policy network, optimized via a PPO variant within formal environments such as Lean. The approach integrates policy rollouts with explicit comparison against an expected policy, jointly orchestrating policy sampling, reward modeling, and multi-step tactic generation—thereby circumventing the reliance on expert-annotated proof trajectories inherent to supervised fine-tuning. Its core contribution is the first introduction of explicit policy rollout versus expected policy comparison into LLM-based theorem proving training, which substantially mitigates exposure bias. Experiments on standard benchmarks demonstrate statistically significant improvements in proof success rate over supervised fine-tuning baselines, validating that reinforcement-driven policy search enhances both reasoning robustness and out-of-distribution generalization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

To take advantage of Large Language Model in theorem formalization and proof, we propose a reinforcement learning framework to iteratively optimize the pretrained LLM by rolling out next tactics and comparing them with the expected ones. The experiment results show that it helps to achieve a higher accuracy compared with directly fine-tuned LLM.

Problem

• Reinforcement learning optimizes theorem proving
• Large Language Model enhances formal theorem accuracy
• Iterative tactic comparison improves proof outcomes

Innovation

• Reinforcement learning optimizes pretrained LLM
• Iterative tactic rollout enhances theorem proof
• Higher accuracy than fine-tuned LLM