Reasoning Models Know What's Important, and Encode It in Their Activations

2026-04-20

AI Summary

This study investigates how language models internally represent the importance of individual steps in multi-step reasoning. Moving beyond surface-level analysis of reasoning chains, the authors train probes on model activations and combine cross-layer activation analysis with human-annotated importance labels to reveal, for the first time, that models encode information about critical reasoning steps in their internal activations prior to generating subsequent steps. Experimental results demonstrate that activation signals predict step importance more accurately than token identities alone, and that this internal representation generalizes across models and is independent of superficial textual features. These findings transcend the limitations of traditional surface-based analyses and offer a novel perspective on the mechanisms underlying model reasoning.

Technology Category

• Knowledge Representation and Reasoning: Reasoning with Beliefs
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Language models often solve complex tasks by generating long reasoning chains, consisting of many steps with varying importance. While some steps are crucial for generating the final answer, others are removable. Determining which steps matter most, and why, remains an open question central to understanding how models process reasoning. We investigate if this question is best approached through model internals or through tokens of the reasoning chain itself. We find that model activations contain more information than tokens for identifying important reasoning steps. Crucially, by training probes on model activations to predict importance, we show that models encode an internal representation of step importance, even prior to the generation of subsequent steps. This internal representation of importance generalizes across models, is distributed across layers, and does not correlate with surface-level features, such as a step's relative position or its length. Our findings suggest that analyzing activations can reveal aspects of reasoning that surface-level approaches fundamentally miss, indicating that reasoning analyses should look into model internals.

Problem

• reasoning chains
• step importance
• model activations
• language models
• internal representations

Innovation

• model activations
• reasoning importance
• probing
• internal representations
• chain-of-thought