State over Tokens: Characterizing the Role of Reasoning Tokens

2025-12-14

AI Summary

This paper addresses the widespread misinterpretation of “reasoning tokens” in large language models (LLMs) as human-like semantic reasoning, clarifying that they are instead the sole persistent externalized representation of internal computation within an otherwise stateless generative process. Method: We introduce the “State over Tokens” (SoT) framework, formally redefining reasoning tokens as symbolic carriers of latent computational states. Integrating generative mechanism analysis, formal modeling of state persistence, and empirical interpretability evaluation, we characterize how reasoning tokens functionally mediate correct output generation. Contribution/Results: Our work refutes the conflation of chain-of-thought prompting with genuine cognitive reasoning, establishes SoT as a principled paradigm for decoding LLM internal states, and opens a new research direction in interpretability grounded in explicit state modeling—thereby advancing foundational understanding of LLM computation and transparency.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) can generate reasoning tokens before their final answer to boost performance on complex tasks. While these sequences seem like human thought processes, empirical evidence reveals that they are not a faithful explanation of the model's actual reasoning process. To address this gap between appearance and function, we introduce the State over Tokens (SoT) conceptual framework. SoT reframes reasoning tokens not as a linguistic narrative, but as an externalized computational state -- the sole persistent information carrier across the model's stateless generation cycles. This explains how the tokens can drive correct reasoning without being a faithful explanation when read as text and surfaces previously overlooked research questions on these tokens. We argue that to truly understand the process that LLMs do, research must move beyond reading the reasoning tokens as text and focus on decoding them as state.

Problem

• Reveals reasoning tokens are not faithful explanations of LLM reasoning processes
• Introduces State over Tokens framework to reframe tokens as computational state
• Proposes decoding reasoning tokens as state rather than reading as text

Innovation

• Reframes reasoning tokens as computational state
• Treats tokens as externalized persistent information carriers
• Focuses on decoding tokens as state not text