Detecting and Characterizing Planning in Language Models

2025-08-25

AI Summary

It remains unclear whether large language models (LLMs) perform genuine forward planning—i.e., pre-selecting goals and generating goal-directed intermediate steps—in multi-step reasoning tasks, or instead rely on token-by-token improvisation. Method: We propose a reproducible, scalable planning detection framework grounded in causal analysis and formalized decision criteria, coupled with a semi-automated annotation pipeline. We evaluate planning behavior across diverse tasks (MBPP code generation and poetry composition) and models (Gemma-2-2B, Claude 3.5 Haiku). Contribution/Results: We find that planning is not a universal capability; rather, models dynamically adapt their reasoning strategies per task (e.g., Gemma-2-2B exhibits pure improvisation in poetry but hybrid planning in MBPP). Instruction tuning does not instantiate planning de novo but fine-tunes pre-existing mechanisms. This work provides the first causal, task-generalizable empirical distinction of internal LLM reasoning strategies, enabling rigorous, interpretable assessment of planning behavior across domains and architectures.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Modern large language models (LLMs) have demonstrated impressive performance across a wide range of multi-step reasoning tasks. Recent work suggests that LLMs may perform planning - selecting a future target token in advance and generating intermediate tokens that lead towards it - rather than merely improvising one token at a time. However, existing studies assume fixed planning horizons and often focus on single prompts or narrow domains. To distinguish planning from improvisation across models and tasks, we present formal and causally grounded criteria for detecting planning and operationalize them as a semi-automated annotation pipeline. We apply this pipeline to both base and instruction-tuned Gemma-2-2B models on the MBPP code generation benchmark and a poem generation task where Claude 3.5 Haiku was previously shown to plan. Our findings show that planning is not universal: unlike Haiku, Gemma-2-2B solves the same poem generation task through improvisation, and on MBPP it switches between planning and improvisation across similar tasks and even successive token predictions. We further show that instruction tuning refines existing planning behaviors in the base model rather than creating them from scratch. Together, these studies provide a reproducible and scalable foundation for mechanistic studies of planning in LLMs.

Problem

• Detecting planning versus improvisation in language models
• Characterizing planning behaviors across different model types
• Establishing reproducible criteria for mechanistic planning studies

Innovation

• Semi-automated annotation pipeline detects planning
• Causally grounded criteria distinguish planning from improvisation
• Applied to Gemma models on code and poem tasks