LM2Alloy: Investigating LLM-Generated Formal Specifications for Automated Test Derivation in Production Software

2026-07-20

AI Summary

Traditional testing approaches struggle to uncover deep logical flaws that violate explicit or implicit constraints, particularly when discrepancies exist between requirements and implementation. This work proposes a novel method that leverages large language models to automatically generate Alloy formal specifications from both requirement documents and source code, using these specifications as an intermediate representation to derive executable test cases. By integrating large language models, formal methods, static analysis, and automated testing, the approach effectively exposes constraint-level defects missed by conventional test generation techniques. Empirical evaluation reveals a real-world vulnerability in the Flipper library and uncovers undocumented implicit abstractions in Cerberus. Moreover, tests derived from code-generated specifications demonstrate superior stability compared to those based solely on requirements.

Abstract

We present an exploratory study on using Large Language Models (LLMs) to generate Alloy formal specifications from both requirements documentation and production source code, and to derive executable test cases from those specifications. We evaluate on two real open-source Python libraries: Flipper, a feature flag management system, and Cerberus, a data validation library. In both cases, the LLM produced workable Alloy specifications and executable tests without any manual correction. For Flipper, our pipeline uncovered a genuine bug that the existing test suite had missed: the library silently accepts duplicate flag names, directly contradicting its documented uniqueness requirement. A direct LLM baseline--generating tests from the same README but skipping the Alloy step--achieved 68% branch coverage yet failed to catch this bug across all three independent runs. This suggests that introducing a formal intermediate representation can surface constraint-level defects that coverage-oriented generation may miss. For Cerberus, the code-derived specification captured an implicit abstraction over sized types that the documentation-derived spec omitted, producing two additional tests. Across both libraries, code-based specifications showed lower variance in test generation (mean SD = 2.15) than documentation-based ones (mean SD = 5.0), though whether this generalises remains an open question. Index Terms--formal specifications, Alloy, large language models, automated testing, specification drift, software validation.

Problem

• formal specifications
• Alloy
• large language models
• automated testing
• specification drift

Innovation

• formal specifications
• Alloy
• large language models
• automated testing
• specification drift