🤖 AI Summary
This study investigates how linguistic information is structurally encoded and evolves across layers in multilingual large language models (LLMs). To this end, we propose the Token-Language Alignment (TLA) analytical framework, which employs both linear and nonlinear probing techniques to perform layer-wise dynamic modeling across all 268 Transformer layers. We find that language identity is highly separable as early as the first layer (accuracy: 76.4±8.2%), and remains approximately linearly separable throughout the entire depth. Crucially, we uncover for the first time that the alignment between language-specific directional representations and vocabulary embeddings is significantly modulated by the language composition of the training corpus—particularly, Chinese-dominant models exhibit a strong “structural imprint effect.” Our work establishes an interpretable, quantifiable analytical paradigm for characterizing the internal representational mechanisms of multilingual LLMs, enabling systematic investigation of cross-lingual representation geometry and its dependence on pretraining data distribution.
📝 Abstract
Multilingual LLMs demonstrate strong performance across diverse languages, yet there has been limited systematic analysis of how language information is structured within their internal representation space and how it emerges across layers. We conduct a comprehensive probing study on six multilingual LLMs, covering all 268 transformer layers, using linear and nonlinear probes together with a new Token--Language Alignment analysis to quantify the layer-wise dynamics and geometric structure of language encoding. Our results show that language information becomes sharply separated in the first transformer block (+76.4$pm$8.2 percentage points from Layer 0 to 1) and remains almost fully linearly separable throughout model depth. We further find that the alignment between language directions and vocabulary embeddings is strongly tied to the language composition of the training data. Notably, Chinese-inclusive models achieve a ZH Match@Peak of 16.43%, whereas English-centric models achieve only 3.90%, revealing a 4.21$ imes$ structural imprinting effect. These findings indicate that multilingual LLMs distinguish languages not by surface script features but by latent representational structures shaped by the training corpus. Our analysis provides practical insights for data composition strategies and fairness in multilingual representation learning. All code and analysis scripts are publicly available at: https://github.com/thisiskorea/How-Language-Directions-Align-with-Token-Geometry-in-Multilingual-LLMs.