Grounded Evaluation and Repair for NL-to-PDDL Problem Generation

2026-09-09

AI Summary

本文针对LLM生成PDDL问题实例时可能出现的任务描述不忠实问题，提出了一种结合生成、检查、验证及迭代修复的端到端NL-to-PDDL流程。

Abstract

Large Language Models (LLMs) have shown promise for translating Natural Language (NL) planning descriptions into PDDL problem instances. However, standard evaluation criteria such as syntactic validity or planner success can substantially overestimate faithfulness to the described task: a generated problem may be parseable and solvable while misrepresenting the intended initial state, goal, object structure, or optimization target. This paper studies an end-to-end NL-to-PDDL pipeline that combines LLM generation, checks in terms of PDDL parsing, planning and validation, a domain-conformance checker, an LLM critic, and iterative repair. Fine-grained repair feedback is constructed from the domain description, the generated problem, the natural language problem description, and operational diagnostics. Reference-based comparisons against curated benchmark PDDL problem descriptions are used for post-hoc benchmark analysis, and these offline checks include renaming-invariant structural matching and semantic equivalence, where domain support is available. Across Planetarium, AutoPlanBench, and curated PDDL~2.1 problems, results show that operational success and benchmark-reference reconstruction can diverge substantially. Results also show that structured repair can be useful, and that PDDL~2.1 remains challenging for reference reconstruction, even when operational success improves.

Problem

• Natural Language
• PDDL
• Problem Generation
• Faithfulness
• Operational Success

Innovation

• end-to-end NL-to-PDDL pipeline
• iterative repair
• domain-conformance checker
• benchmark-reference reconstruction
• structured repair