SpecSyn: LLM-based Synthesis and Refinement of Formal Specifications for Real-world Program Verification

2026-04-23

AI Summary

Current large language models struggle with scalability and the difficulty of evaluating semantic strength when generating formal specifications for large-scale real-world programs. This work proposes SpecSyn, a novel approach that integrates program segmentation, generation of semantically non-equivalent mutants, and a variant discrimination mechanism to iteratively refine specifications, thereby enhancing both precision and completeness. By synergistically combining large language models with static analysis and formal verification techniques, SpecSyn successfully processes 1,071 out of 1,365 open-source programs, achieving a precision exceeding 90% and a recall surpassing 75%, significantly outperforming existing methods.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Program verification is a formal technique to rigorously ensure the correctness and fault-freeness of software systems. However, constructing comprehensive interprocedural specifications for full verification obligations is time-consuming and labor-intensive, giving rise to automated specification generation approaches. Despite the significant advancements in these approaches brought by Large Language Models (LLMs), existing LLM-empowered approaches still suffer from significant limitations: they lack effective strategies for handling sizable input programs, and are typically equipped with no mechanisms to evaluate and guarantee the strength of the generated specifications. The limitations impair their ability to extract precise specifications from real-world complicated programs to support the verification of target properties, thereby hindering the applicability of existing approaches in verification tasks on real-world programs. To remedy this gap, we propose SpecSyn, a novel LLM-based specification generation method. SpecSyn first decomposes the input program into individual segments, which are handled respectively by the subsequent iterative specification generation process. Innovatively, we incorporate into the process a specification refinement mechanism based on semantic-non-equivalent program mutations and variant discrimination, assessing and enhancing the semantic strength of the generated specifications. Extensive experiments show that SpecSyn maintains high precision over 90% and outstanding recall over 75%, significantly outperforming existing LLM-based approaches. In further evaluations, SpecSyn successfully handles 1071 out of 1365 target properties for open-source programs, proving its applicability on real-world program verification tasks.

Problem

• program verification
• formal specifications
• Large Language Models
• specification generation
• real-world programs

Innovation

• specification synthesis
• LLM-based verification
• semantic refinement
• program mutation
• formal specification