How Transformers Learn to Plan via Multi-Token Prediction

2026-04-13

AI Summary

Traditional single-step next-token prediction struggles to capture the global structure inherent in planning-oriented reasoning tasks. This work proposes a multi-token prediction mechanism and demonstrates—through both theoretical analysis and empirical evaluation—that it induces a two-stage backward reasoning process with gradient decoupling properties, yielding clearer training signals that facilitate the learning of interpretable reasoning circuits. Using a simplified two-layer Transformer architecture, the approach is evaluated on synthetic and real-world reasoning benchmarks, including graph pathfinding, the Countdown game, and Boolean satisfiability. Results show that multi-token prediction consistently and significantly outperforms conventional next-token prediction, effectively enhancing the model’s planning capabilities.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While next-token prediction (NTP) has been the standard objective for training language models, it often struggles to capture global structure in reasoning tasks. Multi-token prediction (MTP) has recently emerged as a promising alternative, yet its underlying mechanisms remain poorly understood. In this paper, we study how MTP facilitates reasoning, with a focus on planning. Empirically, we show that MTP consistently outperforms NTP on both synthetic graph path-finding tasks and more realistic reasoning benchmarks, such as Countdown and boolean satisfiability problems. Theoretically, we analyze a simplified two-layer Transformer on a star graph task. We prove that MTP induces a two-stage reverse reasoning process: the model first attends to the end node and then reconstructs the path by tracing intermediate nodes backward. This behavior arises from a gradient decoupling property of MTP, which provides a cleaner training signal compared to NTP. Ultimately, our results highlight how multi-token objectives inherently bias optimization toward robust and interpretable reasoning circuits.

Problem

• multi-token prediction
• reasoning
• planning
• Transformer
• next-token prediction

Innovation

• Multi-Token Prediction
• Reverse Reasoning
• Gradient Decoupling
• Planning
• Interpretable Reasoning