Emergent Manifold Separability during Reasoning in Large Language Models

2026-02-23

AI Summary

This work investigates the dynamic evolution of internal representational geometry in large language models during chain-of-thought (CoT) reasoning. Leveraging Manifold Capacity Theory (MCT), the study quantifies the linear separability of latent representations without requiring probe training, combining compositional Boolean logic tasks with CoT prompting. It reveals that reasoning manifests as a transient “geometric pulse”: conceptual manifolds are initially disentangled into linearly separable subspaces prior to computation and then rapidly compressed. The paper proposes a “dynamic manifold management” mechanism that distinguishes between information retrievability and the geometric structure prepared for computation, identifying a brief window of high separability during reasoning. This highlights the model’s dynamic bandwidth optimization strategy in the residual stream and elucidates the time-varying nature of representational capacity modulation during inference.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Chain-of-Thought (CoT) prompting significantly improves reasoning in Large Language Models, yet the temporal dynamics of the underlying representation geometry remain poorly understood. We investigate these dynamics by applying Manifold Capacity Theory (MCT) to a compositional Boolean logic task, allowing us to quantify the linear separability of latent representations without the confounding factors of probe training. Our analysis reveals that reasoning manifests as a transient geometric pulse, where concept manifolds are untangled into linearly separable subspaces immediately prior to computation and rapidly compressed thereafter. This behavior diverges from standard linear probe accuracy, which remains high long after computation, suggesting a fundamental distinction between information that is merely retrievable and information that is geometrically prepared for processing. We interpret this phenomenon as \emph{Dynamic Manifold Management}, a mechanism where the model dynamically modulates representational capacity to optimize the bandwidth of the residual stream throughout the reasoning chain.

Problem

• Manifold Separability
• Chain-of-Thought
• Representation Geometry
• Large Language Models
• Reasoning Dynamics

Innovation

• Manifold Capacity Theory
• Dynamic Manifold Management
• Chain-of-Thought reasoning
• representation geometry
• linear separability