ProofFlow: A Dependency Graph Approach to Faithful Proof Autoformalization

2025-10-13

AI Summary

Existing formalization methods for natural-language theorems and proofs often lose the semantic content and logical structure of the original arguments. To address this, we propose a formalization framework centered on **structural fidelity**: it explicitly preserves the inference topology of the original proof by constructing a directed acyclic dependency graph over proof steps, augmented with lemma-driven decomposition and stepwise formalization. We introduce ProofScore—a novel composite evaluation metric—and present the first undergraduate-level benchmark dataset comprising 184 problems, designed specifically for structural fidelity assessment. Experiments show our method achieves a ProofScore of 0.545, significantly outperforming full-proof and sentence-by-sentence baselines, establishing new state-of-the-art performance. Our core contributions are: (1) a structure-aware formalization paradigm; (2) an interpretable and verifiable lemma-based formalization pipeline; and (3) the first quantitative evaluation framework targeting structural fidelity.

Technology Category

• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Machine Learning: Structured Learning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Proof autoformalization, the task of translating natural language theorems and proofs into machine-verifiable code, is a critical step for integrating large language models into rigorous mathematical workflows. Current approaches focus on producing executable code, but they frequently fail to preserve the semantic meaning and logical structure of the original human-written argument. To address this, we introduce ProofFlow, a novel pipeline that treats structural fidelity as a primary objective. ProofFlow first constructs a directed acyclic graph (DAG) to map the logical dependencies between proof steps. Then, it employs a novel lemma-based approach to systematically formalize each step as an intermediate lemma, preserving the logical structure of the original argument. To facilitate evaluation, we present a new benchmark of 184 undergraduate-level problems, manually annotated with step-by-step solutions and logical dependency graphs, and introduce ProofScore, a new composite metric to evaluate syntactic correctness, semantic faithfulness, and structural fidelity. Experimental results show our pipeline sets a new state-of-the-art for autoformalization, achieving a ProofScore of 0.545, substantially exceeding baselines like full-proof formalization (0.123), which processes the entire proof at once, and step-proof formalization (0.072), which handles each step independently. Our pipeline, benchmark, and score metric are open-sourced to encourage further progress at https://github.com/Huawei-AI4Math/ProofFlow.

Problem

• Preserving logical structure in proof autoformalization from natural language
• Addressing semantic meaning loss in machine-verifiable code generation
• Improving structural fidelity through dependency graph-based formalization

Innovation

• Constructs dependency graph for logical proof steps
• Employs lemma-based approach for step formalization
• Introduces composite metric evaluating structural fidelity