From Solver Feedback to Faithful Plans: Multi-Role Reinforcement Learning for Symbolic Planning

2026-08-22

AI Summary

研究通过仅使用求解器反馈来学习忠实的自然语言到PDDL形式化方法，提出一个多角色强化学习框架，提高了规划的成功率和准确性。

Abstract

Reliable planning requires converting natural-language instructions into executable symbolic specifications, yet large language models remain brittle without costly PDDL annotations and may exploit solver success in semantically unfaithful ways. We study how to learn faithful natural-language-to-PDDL formalization using only solver feedback, without human-written demonstrations. We propose a solvergrounded multi-role reinforcement learning framework where a single language model acts as an Actor, Judge, and Editor for generation, verification, and repair. The Actor proposes PDDL specifications, the Judge provides a solver-calibrated quality signal, and the Editor performs bounded diagnostic-conditioned refinement. On PlanBench, our method improves average success from 35.5% for LLM+P to 70.8%, achieves 66.3% faithful success, and reduces semantic drift to 6.4%. These results show that organizing solver feedback into generation, verification, and repair roles enables more scalable and faithful annotation-free symbolic planning

Problem

• natural-language-to-PDDL
• solver feedback
• reinforcement learning
• symbolic planning
• faithful formalization

Innovation

• multi-role reinforcement learning
• solver feedback
• PDDL formalization
• generation verification and repair