Planning Anything with Rigor: General-Purpose Zero-Shot Planning with LLM-based Formalized Programming

2024-10-15

arXiv.org

AI Summary

Existing approaches struggle to simultaneously achieve zero-shot generalization and formal solvability for complex planning tasks involving multiple constraints and long-horizon reasoning. Method: This paper proposes the first task-agnostic, example-free framework that uniformly models such planning problems as constrained goal optimization tasks. It introduces an LLM-driven, end-to-end formal programming paradigm integrating commonsense reasoning, programmatic parsing, constraint modeling, optimization solving, and self-consistent verification, augmented by structured output control. Contribution/Results: Evaluated on nine heterogeneous planning benchmarks, the framework achieves pure zero-shot optimal solution generation—attaining average optimal rates of 83.7% on GPT-4o and 86.8% on Claude 3.5 Sonnet—outperforming the strongest baselines by 37.6% and 40.7%, respectively. It fundamentally overcomes the task-specificity and generalization bottlenecks plaguing prior methods.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

While large language models (LLMs) have recently demonstrated strong potential in solving planning problems, there is a trade-off between flexibility and complexity. LLMs, as zero-shot planners themselves, are still not capable of directly generating valid plans for complex planning problems such as multi-constraint or long-horizon tasks. On the other hand, many frameworks aiming to solve complex planning problems often rely on task-specific preparatory efforts, such as task-specific in-context examples and pre-defined critics/verifiers, which limits their cross-task generalization capability. In this paper, we tackle these challenges by observing that the core of many planning problems lies in optimization problems: searching for the optimal solution (best plan) with goals subject to constraints (preconditions and effects of decisions). With LLMs' commonsense, reasoning, and programming capabilities, this opens up the possibilities of a universal LLM-based approach to planning problems. Inspired by this observation, we propose LLMFP, a general-purpose framework that leverages LLMs to capture key information from planning problems and formally formulate and solve them as optimization problems from scratch, with no task-specific examples needed. We apply LLMFP to 9 planning problems, ranging from multi-constraint decision making to multi-step planning problems, and demonstrate that LLMFP achieves on average 83.7% and 86.8% optimal rate across 9 tasks for GPT-4o and Claude 3.5 Sonnet, significantly outperforming the best baseline (direct planning with OpenAI o1-preview) with 37.6% and 40.7% improvements. We also validate components of LLMFP with ablation experiments and analyzed the underlying success and failure reasons. Project page: https://sites.google.com/view/llmfp.

Problem

• Large Language Models
• Complex Planning Problems
• Task-specific Limitations

Innovation

• LLMFP
• Large Language Models
• Zero-Shot Planning