Faithful Autoformalization of Natural Language Assertions

2026-07-14

AI Summary

This work addresses the challenges of ambiguity and validity verification in automatically translating natural language assertions into formal, executable specifications—a task traditionally reliant on error-prone manual effort. The paper proposes Monty, a novel framework that leverages large language models to generate candidate formal assertions and introduces an innovative combination of code-based testing and consistency scoring to automatically select high-quality translations. Evaluated on 541 tasks, Monty achieves up to a 20-percentage-point improvement in average precision over baseline approaches that directly use large language models for translation, substantially enhancing the accuracy and reliability of automated formalization.

Abstract

Formal contracts are essential for software testing and verification, yet writing them remains labor-intensive and error-prone. LLMs offer a promising path toward autoformalization: synthesizing executable assertions from natural-language specifications and thereby bridging the gap between informal developer intent and formal executable specifications. We present Monty: an autoformalization framework for assertions that tackles the challenges of expectations of validity of assertions and ambiguity in natural-language. Our techniques are based on filtering formalizations using a novel conformance score metric and validity scores obtained from testing the code against formalized assertions. We evaluate our approach on 541 assertion-generation tasks derived from 22 collection-like Java classes, and show that our technique produces the ground truth more reliably (improving upto 20 points in precision on average) than when using LLMs naively to translate assertions.

Problem

• autoformalization
• natural language assertions
• formal contracts
• software verification
• LLMs

Innovation

• autoformalization
• conformance score
• validity scoring
• LLM-based assertion synthesis
• formal contracts