🤖 AI Summary
This work investigates how decoder-only large language models transform contextual information into predictive outputs along the depth dimension. By integrating geometric representation analysis with mechanistic interventions, the study reveals—for the first time—that angular components of deep-layer representations encode similarity structures aligned with predictive distributions, while norm components carry non-predictive contextual information. This finding establishes a mechanism-geometric account of the context-to-prediction transformation process. Building upon disentangled representations and an intervenable modeling framework, the research identifies structured geometric properties underlying prediction formation and demonstrates causal, selective control over token-level predictions.
📝 Abstract
We show that decoder-only large language models exhibit a depth-wise transition from context-processing to prediction-forming phases of computation accompanied by a reorganization of representational geometry. Using a unified framework combining geometric analysis with mechanistic intervention, we demonstrate that late-layer representations implement a structured geometric code that enables selective causal control over token prediction. Specifically, angular organization of the representation geometry parametrizes prediction distributional similarity, while representation norms encode context-specific information that does not determine prediction. Together, these results provide a mechanistic-geometric account of the dynamics of transforming context into predictions in LLMs.