Do LLMs Encode Functional Importance of Reasoning Tokens?

2026-01-06

arXiv.org

AI Summary

While long reasoning chains generated by large language models enhance accuracy, they incur substantial computational overhead and obscure the identification of functionally critical reasoning tokens. This work is the first to uncover how the model internally encodes the functional importance of reasoning tokens and introduces a likelihood-preserving greedy pruning strategy that requires no external supervision. By iteratively removing tokens with the least impact on target likelihood, the method produces length-controllable, compact reasoning chains. Integrating attention analysis and knowledge distillation, the approach enables student models trained at equivalent reasoning lengths to outperform existing supervised compression baselines. Moreover, attention scores are shown to effectively predict the optimal pruning order.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models solve complex tasks by generating long reasoning chains, achieving higher accuracy at the cost of increased computational cost and reduced ability to isolate functionally relevant reasoning. Prior work on compact reasoning shortens such chains through probabilistic sampling, heuristics, or supervision from frontier models, but offers limited insight into whether models internally encode token-level functional importance for answer generation. We address this gap diagnostically and propose greedy pruning, a likelihood-preserving deletion procedure that iteratively removes reasoning tokens whose removal minimally degrades model likelihood under a specified objective, yielding length-controlled reasoning chains. We evaluate pruned reasoning in a distillation framework and show that students trained on pruned chains outperform a frontier-model-supervised compression baseline at matched reasoning lengths. Finally, our analysis reveals systematic pruning patterns and shows that attention scores can predict greedy pruning ranks, further suggesting that models encode a nontrivial functional importance structure over reasoning tokens.

Problem

• functional importance
• reasoning tokens
• large language models
• reasoning chains
• token-level importance

Innovation

• greedy pruning
• functional importance
• reasoning compression
• likelihood-preserving deletion
• attention analysis