Make Planning Research Rigorous Again!

2025-05-27

AI Summary

Current LLM-based planning research lacks the methodological rigor accumulated over six decades of classical automated planning, leading to recurring issues such as modeling bias and unreliable evaluation. To address this, we propose— for the first time—a principled integration of classical planning paradigms (PDDL modeling, heuristic search, and standardized benchmark suites) with LLM-based reasoning, establishing a tripartite rigor framework encompassing problem modeling, benchmarking, and reproducible evaluation. We design a hybrid evaluation protocol and cross-paradigm analysis tools to foster community-wide consensus on evaluation standards. Our approach significantly reduces methodological error rates and provides both theoretical foundations and practical guidelines for developing trustworthy, interpretable, and verifiable LLM-based planners.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

In over sixty years since its inception, the field of planning has made significant contributions to both the theory and practice of building planning software that can solve a never-before-seen planning problem. This was done through established practices of rigorous design and evaluation of planning systems. It is our position that this rigor should be applied to the current trend of work on planning with large language models. One way to do so is by correctly incorporating the insights, tools, and data from the automated planning community into the design and evaluation of LLM-based planners. The experience and expertise of the planning community are not just important from a historical perspective; the lessons learned could play a crucial role in accelerating the development of LLM-based planners. This position is particularly important in light of the abundance of recent works that replicate and propagate the same pitfalls that the planning community has encountered and learned from. We believe that avoiding such known pitfalls will contribute greatly to the progress in building LLM-based planners and to planning in general.

Problem

• Applying rigorous planning research methods to LLM-based planners
• Incorporating automated planning tools into LLM planner design
• Avoiding known pitfalls in LLM-based planning systems

Innovation

• Incorporating automated planning insights into LLM-based planners
• Applying rigorous design and evaluation practices
• Avoiding known pitfalls from planning community