Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlation

2025-03-28

AI Summary

This work addresses the geometric paradox between high-dimensional token embeddings (~10³) in large language models (LLMs) and the low-dimensional semantic nature of human language (~10¹). We systematically characterize the dimensional evolution of token representations across Transformer layers: embeddings initially diffuse, then undergo progressive projection, ultimately converging onto a ~10-dimensional semantic submanifold. We introduce “working-space dimension” — a novel intrinsic dimensionality metric — and empirically demonstrate, for the first time, that Transformers implicitly perform inter-layer dimensionality compression; crucially, this compressed dimension exhibits a significant negative correlation with model performance. Our methodology integrates intrinsic dimension estimation (MLE/TwoNN), layer-wise representational correlation analysis, cross-architecture geometric trajectory tracking, and manifold visualization. We validate the universal expansion–contraction pattern across LLaMA, Qwen, and Phi families, and develop a fine-tuning-free diagnostic tool to precisely identify over-parameterization and semantic compression failure.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The geometric evolution of token representations in large language models (LLMs) presents a fundamental paradox: while human language inherently organizes semantic information in low-dimensional spaces ($sim 10^1$ dimensions), modern LLMs employ high-dimensional embeddings ($sim 10^3$ dimensions) processed through Transformer architectures. To resolve this paradox, this work bridges this conceptual gap by developing a geometric framework that tracks token dynamics across Transformers layers. Through layer-wise analysis of intrinsic dimensions across multiple architectures, we reveal an expansion-contraction pattern where tokens diffuse to a"working space"and then progressively project onto lower-dimensional submanifolds. Our finding implies a negative correlation between the working space dimension and parameter-sensitive performance of the LLMs, and indicates that effective models tend to compress tokens into approximately 10-dimensional submanifolds, closely resembling human semantic spaces. This work not only advances LLM interpretability by reframing Transformers layers as projectors that mediate between high-dimensional computation and low-dimensional semantics, but also provides practical tools for model diagnostics that do not rely on task-specific evaluations.

Problem

• Resolve paradox between high-dimensional LLM embeddings and low-dimensional human language semantics
• Analyze token dynamics and intrinsic dimensions across Transformer layers
• Develop geometric framework to link model performance with dimensional reduction

Innovation

• Layer-wise geometric framework tracks token dynamics
• Reveals expansion-contraction pattern in token dimensions
• Links low-dimensional submanifolds to model performance