Fluid Representations in Reasoning Models

2026-02-04

AI Summary

This study investigates the internal mechanisms underlying the strong performance of reasoning language models on abstract tasks, with a focus on how they dynamically construct and refine structured representations. Using the QwQ-32B model’s reasoning process on the Mystery Blocksworld task as a case study, the work introduces the concept of “fluid reasoning representations,” revealing that the model incrementally encodes abstract structures—rather than concrete symbols—through contextual interaction. Through a combination of mechanistic interpretability analyses, representation interventions, symbol substitution experiments, and structural task evaluations, the study demonstrates that these dynamic representations causally contribute to reasoning performance: injecting optimized representations significantly improves accuracy, and symbolic representations can effectively replace entangled encodings with minimal performance loss, thereby elucidating a key mechanism for enhancing reasoning capabilities.

Technology Category

• Knowledge Representation and Reasoning: Argumentation
• Cognitive Modeling & Cognitive Systems: Symbolic Representations
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Reasoning language models, which generate long chains of thought, dramatically outperform non-reasoning language models on abstract problems. However, the internal model mechanisms that allow this superior performance remain poorly understood. We present a mechanistic analysis of how QwQ-32B - a model specifically trained to produce extensive reasoning traces - process abstract structural information. On Mystery Blocksworld - a semantically obfuscated planning domain - we find that QwQ-32B gradually improves its internal representation of actions and concepts during reasoning. The model develops abstract encodings that focus on structure rather than specific action names. Through steering experiments, we establish causal evidence that these adaptations improve problem solving: injecting refined representations from successful traces boosts accuracy, while symbolic representations can replace many obfuscated encodings with minimal performance loss. We find that one of the factors driving reasoning model performance is in-context refinement of token representations, which we dub Fluid Reasoning Representations.

Problem

• reasoning language models
• abstract reasoning
• internal representations
• structural information
• fluid representations

Innovation

• Fluid Reasoning Representations
• mechanistic interpretability
• abstract reasoning
• in-context representation refinement
• reasoning language models