Code Driven Planning with Domain-Adaptive Critic

2025-09-23

AI Summary

To address the misalignment between general knowledge encoded in large language models (LLMs) and task-specific environmental requirements—leading to inaccurate planning when LLMs serve as AI agent planners—this paper proposes an efficient long-horizon planning framework. It replaces frequent LLM queries with executable, high-level code-based planning programs, significantly reducing computational overhead. A trainable, domain-adaptive critic is introduced to evaluate and filter candidate programs based on long-term reward consistency, thereby mitigating the limitations of myopic feedback. The core innovation lies in decoupling planning into two distinct phases—program generation and critic-based evaluation—and enabling environment-adaptive training of the critic. Evaluated on ALFWorld, NetHack, and StarCraft II construction tasks, the framework achieves an average success rate improvement of 23.33% while reducing LLM query volume by 91.27%.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Multiagent Systems: Multiagent Planning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Models (LLMs) have been widely adopted as task planners for AI agents in sequential decision-making problems, leveraging their extensive world knowledge. However, the gap between their general knowledge and environment-specific requirements often leads to inaccurate plans. To address this, existing approaches rely on frequent LLM queries to iteratively refine plans based on immediate environmental feedback, which incurs substantial query costs. However, this refinement is typically guided by short-term environmental feedback, limiting LLMs from developing plans aligned with long-term rewards. We propose Code Driven Planning with Domain-Adaptive Critic (CoPiC). Instead of relying on frequent queries, CoPiC employs LLMs to generate a diverse set of high-level planning programs, which iteratively produce and refine candidate plans. A trained domain-adaptive critic then evaluates these candidates and selects the one most aligned with long-term rewards for execution. Using high-level planning programs as planner and domain-adaptive critic as estimator, CoPiC improves planning while significantly reducing query costs. Results in ALFWorld, NetHack, and StarCraft II Unit Building show that CoPiC outperforms advanced LLM-based baselines, AdaPlanner and Reflexion, achieving an average (1) 23.33% improvement in success rate and (2) 91.27% reduction in query costs.

Problem

• Bridging the gap between LLMs' general knowledge and environment-specific planning requirements
• Reducing substantial query costs from frequent LLM interactions in iterative planning
• Improving plan quality by aligning decisions with long-term rewards rather than short-term feedback

Innovation

• Generates diverse high-level planning programs
• Uses domain-adaptive critic for long-term reward evaluation
• Reduces LLM queries by refining candidate plans iteratively