Translating Natural Language to Strategic Temporal Specifications via LLMs

2026-06-29

AI Summary

This work addresses the significant challenge of automatically translating natural language descriptions of multi-agent strategic requirements into formal specifications such as ATL or ATL*, a task currently hindered by reliance on expert knowledge and the absence of automated tools. We propose the first NL-to-ATL/ATL* generation framework tailored for multi-agent strategic logic, introducing and open-sourcing the first expert-annotated dataset to enable supervised fine-tuning of open-source large language models with 3–7B parameters. Our approach integrates a semantic validation mechanism with existing model checkers to ensure correctness. Experimental results demonstrate that the fine-tuned models achieve a semantic accuracy of 0.84, statistically on par with the strongest few-shot closed-source API baseline (0.86), thereby substantially lowering the barrier for non-expert users to generate verifiable formal specifications.

Abstract

A rigorous formalization of system requirements is a fundamental prerequisite for the verification of Multi-Agent Systems (MAS). However, writing correct formal specifications is well known as an error-prone, time-consuming, and expertise-intensive task. This difficulty is further accentuated in MAS, where requirements must capture strategic abilities and temporal objectives. At present, there is no established methodology for deriving MAS specifications from natural language. We present a framework for translating Natural Language descriptions of strategic requirements into well-formed ATL/ATL* formulas using Large Language Models (LLMs). Since no available dataset supports supervised learning for the NL-to-ATL/ATL* translation task, we create and curate a novel expert-validated dataset, employed for training and evaluating fine-tuned models. On a held-out test set, evaluated under the LLM judge that best agrees with expert annotations, in-domain fine-tuning of small open-weight models (3 - 7B parameters) matches strong few-shot proprietary API baselines. Our best fine-tuned system reaches 0.84 semantic accuracy, statistically on par with 0.86 for the strongest few-shot proprietary baseline, while keeping requirements on-premises. We further find that judge reliability is inverse to generator strength. The open-weight Llama-3.3-70B tracks human verdicts most closely, whereas the strongest proprietary models are the least reliable judges, over-rejecting faithful paraphrases of the reference. To assess the practical applicability of the generated specifications, we embed our tool to an existing strategic logics model checker, enabling non-expert users to specify strategic properties in natural language.

Problem

• Natural Language to Formal Specification
• Multi-Agent Systems
• Strategic Temporal Logic
• ATL/ATL*
• Requirement Formalization

Innovation

• Natural Language to Formal Specification
• ATL/ATL*
• Large Language Models
• Multi-Agent Systems
• Strategic Temporal Logic