Theory-Scale Auto-Formalization of Logics for Computer Science

2026-06-24

AI Summary

This work proposes a theory-level collaborative formalization framework to address the challenges of consistency, fidelity, and scalability in large-scale automated formalization of definitions, lemmas, and theorems. The approach integrates concept graph modeling, formal signature planning, counterexample-driven completion, expert fidelity review, and an automated evaluation engine to construct LCS-Bench—the first theory-level benchmark for formalization. Comprising 327 textbook topics, 4,076 Lean statements, and over 85K lines of code, LCS-Bench demonstrates substantial difficulty: evaluations across 14 state-of-the-art models reveal that even the best-performing system achieves only a 20.1% accuracy in automatic formalization, underscoring the benchmark’s value as a rigorous platform for evaluating advanced formalization capabilities.

Abstract

Auto-formalization is critical for scalable formal verification, but existing progress largely focuses on isolated statements, while theory-scale auto-formalization, which coherently translates hundreds of interdependent definitions, lemmas, and theorems, remains open due to challenges in consistency, faithfulness, scalability, and correctness. In this paper, we introduce LCS-Bench, a stand-alone, theory-scale benchmark based on Logics for Computer Science. LCS-Bench is built through a novel semi-automated agentic pipeline that leverages concept graphs, formal signature planning, issue tracking, sorry-filling with counter-example search, complemented by faithfulness review from human experts. The resulting artifact covers 327 textbook items, over 4,076 Lean declarations, and more than 85K lines of Lean code. The dataset supports broad evaluation through a data engine that automatically derives five tracks of evaluation benchmarks, measuring different aspects of auto-formalization and theorem-proving capabilities. We also introduce a novel evaluation protocol featuring definitional equivalence checkers, enabling more fine-grained and faithful assessment. Through extensive evaluation on 14 models, we demonstrate that (1) LCS-Bench is of high quality, consistent, and faithful; (2) the benchmark is challenging, with state-of-the-art models achieving only 20.1% on auto-formalization tasks; and (3) our analysis reveals key findings regarding theory-scale auto-formalization and suggests promising directions for future work.

Problem

• auto-formalization
• theory-scale
• formal verification
• consistency
• faithfulness

Innovation

• theory-scale auto-formalization
• LCS-Bench
• semi-automated agentic pipeline
• definitional equivalence checking
• formal verification