FM-Agent: Scaling Formal Methods to Large Systems via LLM-Based Hoare-Style Reasoning

2026-04-13

AI Summary

This work addresses the challenges of formally verifying code generated by large language models (LLMs), particularly the heavy burden of writing formal specifications and the difficulty of adapting verification techniques to black-box code. The authors propose FM-Agent, a novel framework that introduces the first automated, compositional formal verification approach tailored for large-scale systems. Leveraging a top-down strategy, FM-Agent employs LLMs to automatically infer function-level natural-language specifications from calling contexts that align with developer intent. It extends Hoare logic to support compositional reasoning with natural-language specifications and concurrently generates test cases to localize and explain vulnerabilities. Evaluated on systems up to 143,000 lines of code, the method successfully verified these within two days and uncovered 522 previously unknown critical bugs, including those causing crashes and execution errors.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

LLM-assisted software development has become increasingly prevalent, and can generate large-scale systems, such as compilers. It becomes crucial to strengthen the correctness of the generated code. However, automated reasoning for large-scale systems remains challenging due to code complexity. Hoare logic offers an approach to decomposing a large system into smaller components and reasoning about them separately (i.e., compositional reasoning). However, existing works still struggle to scale, because Hoare logic requires writing formal specifications for each function, imposing a heavy human burden. The problem is exacerbated when code is generated by LLMs, as developers lack a deep understanding of each function's expected behavior. This paper presents FM-Agent, the first framework that realizes automated compositional reasoning for large-scale systems. Leveraging LLMs, FM-Agent introduces a top-down paradigm to automatically generate function-level specifications. Specifically, FM-Agent derives the specification of a function from how its callers expect the function to behave, so the generated specifications can reflect the developer's intent of a function even if the implementation is buggy. Developers' intent is usually expressed in natural language, while existing verifiers only support formulas. Therefore, FM-Agent generalizes Hoare-style inference to reason about functions against natural-language specifications. Finally, to confirm bug existence and explain bug causes, FM-Agent automatically generates test cases to trigger potential bugs. In our evaluation, FM-Agent successfully reasons about large-scale systems within 2 days, each of which has up to 143k LoC. These systems have already been tested by their developers, but FM-Agent still finds 522 newly discovered bugs. These bugs can cause serious consequences, including system crashes and incorrect execution results.

Problem

• Formal Methods
• Compositional Reasoning
• Hoare Logic
• Large Language Models
• Specification Generation

Innovation

• compositional reasoning
• Hoare logic
• LLM-based specification generation
• natural-language specifications
• automated verification