FormalScience: Scalable Human-in-the-Loop Autoformalisation of Science with Agentic Code Generation in Lean

2026-04-24

AI Summary

This work addresses the challenge of automatically translating informal physical reasoning into formally verifiable Lean4 proofs, which is hindered by misalignment between domain-specific notation and formal semantics. The authors propose a domain-agnostic, human-in-the-loop agent framework that enables experts without formalization experience to efficiently generate syntactically correct and semantically coherent formal proofs. They provide the first systematic characterization of semantic drift in physics formalization, introducing concepts such as “notation collapse” and “abstraction lifting,” and construct FormalPhysics—a high-complexity dataset that is fully formally valid. By integrating multi-stage agents, zero-shot prompting, error-feedback self-refinement, and an interactive UI, their approach achieves 100% formal validity on 200 university-level physics problems, substantially outperforming existing mathematical formalization benchmarks. The code and system are open-sourced to support extension across scientific domains.

Technology Category

• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Computer Vision: Low Level & Physics-based Vision
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Security and Privacy: Data transparency and provenance

Abstract

Formalising informal mathematical reasoning into formally verifiable code is a significant challenge for large language models. In scientific fields such as physics, domain-specific machinery (\textit{e.g.} Dirac notation, vector calculus) imposes additional formalisation challenges that modern LLMs and agentic approaches have yet to tackle. To aid autoformalisation in scientific domains, we present FormalScience; a domain-agnostic human-in-the-loop agentic pipeline that enables a single domain expert (without deep formal language experience) to produce \textit{syntactically correct} and \textit{semantically aligned} formal proofs of informal reasoning for low economic cost. Applying FormalScience to physics, we construct FormalPhysics, a dataset of 200 university-level (LaTeX) physics problems and solutions (primarily quantum mechanics and electromagnetism), along with their Lean4 formal representations. Compared to existing formal math benchmarks, FormalPhysics achieves perfect formal validity and exhibits greater statement complexity. We evaluate open-source models and proprietary systems on a statement autoformalisation task on our dataset via zero-shot prompting, self-refinement with error feedback, and a novel multi-stage agentic approach, and explore autoformalisation limitations in modern LLM-based approaches. We provide the first systematic characterisation of semantic drift in physics autoformalisation in terms of concepts such as notational collapse and abstraction elevation which reveals what formal language verifies when full semantic preservation is unattainable. We release the codebase together with an interactive UI-based FormalScience system which facilitates autoformalisation and theorem proving in scientific domains beyond physics.https://github.com/jmeadows17/formal-science

Problem

• autoformalisation
• scientific reasoning
• formal verification
• domain-specific notation
• semantic alignment

Innovation

• autoformalisation
• human-in-the-loop
• agentic code generation
• semantic drift
• FormalPhysics