Evaluating LLM-Generated ACSL Annotations for Formal Verification

2026-02-14

AI Summary

Automatically generating accurate and verifiable ACSL (ANSI/ISO C Specification Language) specifications remains a key challenge in C program verification. This work presents the first systematic evaluation of the verifiability and stability of ACSL annotations produced by rule-based scripts, the Frama-C RTE plugin, and three large language models—DeepSeek-V3.2, GPT-5.2, and OLMo 3.1 32B Instruct—under a fully automatic, non-interactive, and learning-free setting. Using a unified verification framework based on the Frama-C WP plugin coupled with multiple SMT solvers, the study provides new empirical evidence on the quality of automated ACSL generation, solver sensitivity, and proof stability, thereby revealing both the strengths and limitations of current approaches.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Formal specifications are crucial for building verifiable and dependable software systems, yet generating accurate and verifiable specifications for real-world C programs remains challenging. This paper empirically evaluates the extent to which formal-analysis tools can automatically generate and verify ACSL specifications without human or learning-based assistance. We conduct a controlled study on a recently released dataset of 506 C programs, repurposing it from interactive, developer-driven workflows to an automated evaluation setting. Five ACSL generation systems are compared: a rule-based Python script, Frama-C's RTE plugin, and three large language models--DeepSeek-V3.2, GPT-5.2, and OLMo 3.1 32B Instruct. All generated specifications are verified under identical conditions using the Frama-C WP plugin powered by multiple SMT solvers, allowing a direct comparison of annotation quality, solver sensitivity, and proof stability. Our results provide new empirical evidence on the capabilities and limitations of automated ACSL generation, complementing prior survey-based work.

Problem

• ACSL
• Formal Verification
• LLM
• C Programs
• Specification Generation

Innovation

• ACSL
• formal verification
• LLM-generated specifications
• automated evaluation
• Frama-C