An Empirical Study of LLM-Generated Specifications for VeriFast

2026-06-24

AI Summary

This study addresses the high specification burden of static verification tools like VeriFast, which, despite their ability to verify complex heap-manipulating programs using separation logic, require costly manual annotation. For the first time, this work systematically evaluates the capability of large language models (LLMs) to automatically generate C function specifications for VeriFast, examining ten prominent LLMs, eight prompting strategies, and three input formats through both quantitative and qualitative analyses across two experimental phases. Results show that LLM-generated specifications achieve over 91% functional behavioral consistency and a 31.4% verification success rate; notably, 94% of failures stem from insufficient domain-specific knowledge of VeriFast. The findings highlight domain adaptation as a critical bottleneck and propose effective strategies to improve verification success rates.

Abstract

Static verification tools can assure industrial scale software, but require significant human labor to write specifications. This is particularly true of static verifiers based on separation logic (SL verifiers), which excel at verifying heapmanipulating programs, but require many complex auxiliary specifications to reason about heap structure. Recent work applies large language models (LLMs) to generate code, tests, and proofs, including specifications for verifiers, but mostly targeting non-SL verifiers. To address this gap, this paper thoroughly evaluates how well LLMs perform when prompted to generate specifications for verifying 303 C functions with the SL verifier VeriFast. We explored eight prompting approaches, ten LLMs, and three input types in two stages. Quantitative and qualitative analyses are used to assess the LLM-generated code and specifications for functional behavior, verifiability and errors. The results show that LLMs preserve functional behavior in source code and specifications (both over 91%), but achieve modest verification success (31.4%). Using Gemini 2.5 Pro and providing formal contracts lead to higher success rates in our setting. Moreover, most errors (94%) come from LLMs' mistakes in the domainspecific knowledge of SL verifiers such as VeriFast. These findings provide guidance for optimizing LLM-generated specifications for SL verifiers.

Problem

• static verification
• separation logic
• specification generation
• LLM
• VeriFast

Innovation

• Large Language Models
• Separation Logic
• Static Verification
• Specification Generation
• VeriFast