What We Talk About When We Talk About LLM Planning: Evidence for Two Distinct Planning Abilities

2026-07-13

AI Summary

This study addresses the inconsistent performance of large language models (LLMs) on planning tasks, challenging the conventional attribution to task difficulty alone by uncovering an underlying multidimensional competence structure. Leveraging the ACP Bench-Hard benchmark and applying multidimensional item response theory under varying reasoning budgets, the work reveals for the first time that planning ability comprises two distinct dimensions: operational reasoning and structural enumeration. The findings demonstrate that operational reasoning improves markedly with increased model scale and reasoning length, whereas structural enumeration remains relatively stable across conditions. This insight transcends the limitations of holistic performance evaluation in planning tasks and establishes a novel paradigm for fine-grained analysis and targeted enhancement of LLM planning capabilities.

Abstract

When LLMs exhibit uneven performance across planning tasks, these gaps are often attributed to task difficulty. We argue that this explanation is incomplete, as task-level variation may reflect distinct latent planning competencies rather than differences along a single ability spectrum. We study this question on ACPBench-Hard by evaluating multiple LLM families under varying test-time reasoning budgets and applying a multidimensional item response theory model to uncover the latent competency structure underlying LLM planning. The analysis reveals two principal dimensions that shape planning performance: operational reasoning, the ability to evaluate local action applicability and immediate state transitions, and structural enumeration, the ability to reason about goal reachability and landmark structure. Operational reasoning improving under model scaling and longer reasoning traces, while structural enumeration remains comparatively insensitive. Our findings motivate competency-level evaluation of LLM planning, shifting the focus from whether models improve overall to which planning competencies improve, under what conditions, and why.

Problem

• LLM planning
• planning abilities
• task difficulty
• latent competencies
• multidimensional evaluation

Innovation

• operational reasoning
• structural enumeration
• multidimensional item response theory
• LLM planning
• competency-level evaluation