Laurel: Unblocking Automated Verification with Large Language Models

2024-05-27

AI Summary

In program verification, SMT solvers frequently fail due to missing critical assertions, necessitating manual assertion hints and substantially increasing verification overhead. This paper proposes an LLM-based automated assertion completion method: (1) it precisely localizes assertion gaps using SMT error messages and introduces placeholder tokens; (2) it defines a code-level proof similarity metric to enable context-aware example retrieval; and (3) it integrates domain-specific prompt engineering with SMT feedback-driven iterative refinement. Evaluated on the DafnyGym benchmark, our approach generates over 56.6% of required assertions in a single attempt, significantly improving automated verification success rates. Our key contributions are the first integration of error-driven localization, proof-aware retrieval, and verification-feedback closed-loop optimization into an LLM-assisted assertion generation framework.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Program verifiers such as Dafny automate proofs by outsourcing them to an SMT solver. This automation is not perfect, however, and the solver often requires hints in the form of assertions, creating a burden for the proof engineer. In this paper, we propose Laurel, a tool that alleviates this burden by automatically generating assertions using large language models (LLMs). To improve the success rate of LLMs in this task, we design two domain-specific prompting techniques. First, we help the LLM determine the location of the missing assertion by analyzing the verifier's error message and inserting an assertion placeholder at that location. Second, we provide the LLM with example assertions from the same codebase, which we select based on a new proof similarity metric. We evaluate our techniques on our new benchmark DafnyGym, a dataset of complex lemmas we extracted from three real-world Dafny codebases. Our evaluation shows that Laurel is able to generate over 56.6% of the required assertions given only a few attempts, making LLMs an affordable tool for unblocking program verifiers without human intervention.

Problem

• Automated program verification often requires manual assertion hints.
• Laurel uses LLMs to generate assertions, reducing human intervention.
• Domain-specific prompting improves LLM success in assertion generation.

Innovation

• LLMs generate assertions for program verifiers
• Domain-specific prompts improve assertion accuracy
• Proof similarity metric guides assertion examples