KBSpec: LLM-driven Formal Specification Generation with Evolving Domain Knowledge Base

2026-06-19

AI Summary

This work addresses the challenge that large language models (LLMs) often produce syntactically incorrect or unverifiable Java Modeling Language (JML) specifications due to insufficient domain-specific training data. To overcome this limitation, the authors propose a self-evolving knowledge base mechanism that requires neither fine-tuning nor labeled data. By dynamically integrating external documentation with feedback from an internal verifier, the method continuously constructs and updates a domain-specific knowledge repository to enhance specification quality. Evaluated across three prominent LLMs, the approach significantly improves verification pass rates by 10–25% compared to existing techniques and generates specifications with higher completeness, demonstrating effective unsupervised, continual improvement in formal specification generation.

Abstract

Automated formal specification generation is a key step towards program understanding and formal verification. Recently, due to the success of large language models (LLMs) in code generation, researchers have made early attempts to adopt LLMs for generating formal specifications. However, the lack of formal specification language corpora in the wild often makes LLMs fail to generate syntactically correct and semantically verifiable specifications. To mitigate this gap, we propose KBSpec, which augments LLMs with dual-source knowledge of formal specification language: external knowledge from official documentation, and internal knowledge distilled from verifier feedback on LLM-generated specifications. KBSpec maintains a self-evolving knowledge base that is continuously updated from successful generation and repair trajectories, without any LLM parameter tuning or labeled training data. We evaluate KBSpec on Java Modeling Language (JML) specification generation with three LLM backends, and results show that KBSpec improves verification pass rates by 10-25% over state-of-the-art LLM-based approaches, while producing the largest number of high-completeness specifications.

Problem

• formal specification generation
• large language models
• program verification
• JML
• domain knowledge

Innovation

• formal specification generation
• large language models
• knowledge base
• self-evolving system
• program verification