Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization

2026-06-29

AI Summary

Current approaches to generating Lean formal statements from natural language rely solely on compilation success rates, which often fail to ensure semantic faithfulness, leading to issues such as omitted premises, incorrect domain specifications, or trivial propositions. This work proposes a novel evaluation paradigm centered on “consensus faithfulness,” introducing a benchmark dataset of 400 graduate-level mathematical problems. Through an integrated methodology combining Lean compilation checks, multi-model semantic evaluations, expert calibration, and a 2³ factorial experiment, the study systematically identifies key factors affecting faithfulness. Experiments reveal that even the best tool-augmented agent achieves only 60.5% consensus faithfulness despite an 89.5% compilation rate, and manual auditing confirms this metric’s high conservativeness, highlighting a substantial gap between syntactic compilability and genuine semantic accuracy.

Abstract

Theorem-proving benchmarks evaluate proof search against fixed formal statements, but natural-language-to-Lean formalization must generate the formal statement itself. In this setting, compilation is only a validity check: a Lean declaration may type-check while omitting hypotheses, changing domains, or expressing a vacuous claim. We study faithful statement formalization as both an evaluation problem and a bottleneck-attribution problem. On a 400-entry graduate-level benchmark spanning real analysis, complex analysis, topology, and algebra, our protocol combines Lean compilation, cross-model semantic judging, and human expert calibration. The resulting picture is different from compile-rate evaluation: a full tool-augmented agent reaches 89.5% compilation but only 60.5% consensus faithfulness, exposing a 29.0-point compile-pass but consensus-unfaithful gap. Targeted human audits support the metric as a conservative decision boundary: across available case-level audits, 96.0% of consensus-positive outputs are human-confirmed faithful, while 82.4% of compile-pass consensus-negative outputs are human-confirmed semantic failures. Under this metric, existing one-shot formalizer models and prover-oriented Lean models remain low, suggesting that formal validity, proof-oriented Lean competence, and faithful statement generation should be reported separately. We then use a full $2^3$ factorial design to decompose three recurring interventions in formalization pipelines: parametric expert drafting, Mathlib/context search, and Lean elaboration feedback. Elaboration feedback is the largest validity intervention, but it also exposes a larger compile-pass semantic-failure bucket; search mainly improves grounding and selectivity; and fine-tuned drafting is largely substitutable in this tool stack once feedback and grounding are available.

Problem

• faithful formalization
• natural-language-to-Lean
• theorem proving
• semantic correctness
• formal statement generation

Innovation

• faithful formalization
• natural-language-to-Lean
• semantic faithfulness evaluation
• factorial intervention analysis
• theorem proving benchmark