Sorries Are Not the Hard Part: An Expert-Review Case Study of a Semi-Autonomous Formalization

2026-06-11

AI Summary

While current large language models can automatically fill proof holes (i.e., eliminate 'sorries') in interactive theorem proving, their generated formalizations often fail expert review due to ill-conceived definitions, insufficiently general theorems, or suboptimal API design. This work presents a semi-autonomous formalization of Grothendieck’s vanishing theorem as a case study and introduces expert review as a central criterion for evaluating the quality of automated formalizations. By integrating large language model assistance, interactive proving, and an iterative refactoring-compression pipeline, the study systematically assesses the high-level design usability of automatically generated content. The findings reveal that measuring success solely by 'sorry' closure is markedly inadequate; expert-driven refactoring substantially improves formalization quality, underscoring the critical role of expert acceptability in evaluating automated formalization efforts.

Abstract

Large language models can often close proof gaps in interactive theorem provers, but a verified theorem is not the same thing as a reusable library contribution. We study this distinction through a detailed case study: a semi-autonomous formalization of Grothendieck's vanishing theorem. The initial version compiles with no sorries, but an expert review found serious problems in definitions, theorem generality, file organization, and the API. We then ran a review-driven refactor and compression process and obtained a second expert review. The before-and-after comparison shows a sharp split: agents adapted well to local, mechanically checkable feedback, but remained weak at choosing definitions and designing APIs. We argue that autoformalization should be evaluated not only by closed sorries, but by whether the resulting formalization survives expert review.

Problem

• autoformalization
• interactive theorem proving
• expert review
• formalization quality
• large language models

Innovation

• autoformalization
• expert review
• interactive theorem proving
• large language models
• formal verification