Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization

2025-10-15

AI Summary

Current large language model (LLM) inference lacks interpretability, and conventional reinforcement learning (RL) employs global, uniform credit assignment—failing to distinguish critical reasoning steps from routine ones. Method: We uncover an intrinsic “pre-planning–anchoring” reasoning mechanism in LLMs and propose two novel metrics—window-averaged attention distance and future attention influence—to formally characterize its rhythmic structure. We further design a structure-aware, fine-grained policy learning framework integrating head-level local/global attention analysis, zigzag pattern detection, and influence quantification. Contribution/Results: Our approach validates three novel RL policies across diverse reasoning tasks, achieving significant performance gains. This work advances LLM inference optimization toward transparency, mechanistic understanding, and dynamic, step-aware credit assignment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The reasoning pattern of Large language models (LLMs) remains opaque, and Reinforcement learning (RL) typically applies uniform credit across an entire generation, blurring the distinction between pivotal and routine steps. This work positions attention as a privileged substrate that renders the internal logic of LLMs legible, not merely as a byproduct of computation, but as a mechanistic blueprint of reasoning itself. We first distinguish attention heads between locally and globally focused information processing and reveal that locally focused heads produce a sawtooth pattern near the diagonal indicating phrasal chunks, while globally focused heads expose tokens that exert broad downstream influence over future tokens. We formalize these with two metrics: 1) Windowed Average Attention Distance, which measures the extent of backward attention within a clipped window; 2) Future Attention Influence, which quantifies a token's global importance as the average attention it receives from subsequent tokens. Taken together, these signals reveal a recurring preplan-and-anchor mechanism, where the model first performs a long-range contextual reference to generate an introductory token, which is immediately followed by or coincides with a semantic anchor token that organizes subsequent reasoning. Leveraging these insights, we introduce three novel RL strategies that dynamically perform targeted credit assignment to critical nodes (preplan tokens, anchor tokens, and their temporal coupling) and show consistent performance gains across various reasoning tasks. By aligning optimization with the model's intrinsic reasoning rhythm, we aim to transform opaque optimization into an actionable structure-aware process, hoping to offer a potential step toward more transparent and effective optimization of LLM reasoning.

Problem

• Unveils attention mechanisms as blueprint for LLM reasoning patterns
• Identifies preplan-anchor rhythm through local and global attention analysis
• Develops targeted RL strategies for structure-aware credit assignment optimization

Innovation

• Attention heads reveal preplan-anchor reasoning mechanism
• Windowed Average Attention Distance measures backward attention
• Future Attention Influence quantifies token global importance