PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models

2026-05-20

AI Summary

Existing planning benchmark datasets suffer from limitations in scalability, controllability, and automated verifiability, hindering effective evaluation or training of large language models (LLMs) on complex planning tasks. This work proposes the first constraint-driven synthetic framework that leverages a structured taxonomy of task types and constraint families to generate planning problems on demand—offering controllable difficulty levels, diverse scenarios, and built-in automatic verifiability. By integrating a constraint-driven data synthesis pipeline, quality filtering mechanisms, and instance-level verification checklists, the framework shifts planning data construction from static collection to dynamic generation. Experiments reveal that current LLMs exhibit limited planning capabilities under coupled constraints, whereas reinforcement learning trained on our dataset substantially improves their performance on unseen tasks and general instruction following.

Abstract

Planning is a fundamental capability for large language models (LLMs) because such complex tasks require models to coordinate goals, constraints, resources, and long-term consequences into executable and verifiable solutions. Existing planning benchmarks, however, usually treat planning data as fixed collections of instances rather than controllable generation targets. This limits scenario coverage, ties difficulty to surface-level proxies rather than structural sources, and offers limited support for scalable generation, automatic verification, or planning-oriented training. We introduce PlanningBench, a framework for generating scalable, diverse, and verifiable planning data for both evaluation and training. PlanningBench starts from real planning scenarios and abstracts practical workflows into a structured taxonomy of more than 30 task types, subtasks, constraint families, and difficulty factors. Guided by this taxonomy, a constraint-driven synthesis pipeline instantiates self-contained planning problems with adaptive difficulty control, quality filtering, and instance-level verification checklists. This shifts planning data construction from fixed benchmark collection to controllable generation while preserving realistic task grounding. We use PlanningBench to evaluate open-source and closed-source frontier LLMs, and find that current models still struggle to produce complete solutions under coupled constraints. Beyond evaluation, reinforcement learning on verified PlanningBench data improves performance on unseen planning benchmarks and broader instruction-following tasks. Further analysis suggests that determinate or well-specified optimal solutions provide clearer reward signals and more stable training dynamics. Overall, PlanningBench provides a controllable source of planning data for diagnosing and improving generalizable planning abilities in LLMs.

Problem

• planning benchmarks
• scalable data generation
• verifiable planning
• LLM evaluation
• constraint-driven synthesis

Innovation

• controllable data generation
• constraint-driven synthesis
• verifiable planning
• structured taxonomy
• planning-oriented training