Layer-wise Representation Dynamics: An Empirical Investigation Across Embedders and Base LLMs

2026-05-12

AI Summary

Existing analyses of representational dynamics across layers in language models are often confined to a single dimension, limiting their ability to fully characterize the complex evolution of internal representations. This work proposes the Layer Representation Dynamics (LRD) framework, which establishes a unified multidimensional perspective by integrating the Frenet frame, Neighborhood Retention Score (NRS), and Graph-Filtered Mutual Information (GFMI) to quantify, respectively, global subspace motion, local structural preservation, and alignment with the final-layer representation. Experiments across 31 models and 30 MTEB tasks demonstrate that LRD uncovers architectural and task-related differences invisible to final-layer-only analyses. Notably, the distance between initial and final layers ($d_{0,L}$) strongly correlates with downstream performance, and GFMI-guided layer pruning consistently outperforms random strategies under 15%–20% computational budgets, achieving median-optimal performance. The framework thus enables effective unsupervised model selection and efficient inference.

Abstract

Hidden states change substantially across the layers of modern language models, but most layer-wise analyses focus on one aspect of that change. We propose Layer-wise Representation Dynamics (LRD), a framework with three layer-wise measurement families: Frenet (Grassmann speed and curvature) for global subspace motion, Neighborhood Retention Score (NRS) for local nearest-neighbor retention, and Graph Filtration Mutual Information (GFMI) for alignment with the final layer. Applying LRD to 31 models (encoder-based and decoder-based embedders, plus base LLMs) on 30 MTEB tasks reveals architectural and task-level differences that are not apparent from final-layer representations alone. We then use LRD for two applications: label-free model selection and inference-time layer pruning. For selection, all three model-level scores correlate positively with downstream MTEB performance, with end-to-end subspace displacement (d_{0,L}) the strongest, and the same direction holds on a smaller base-LLM MMLU panel. For pruning, GFMI is the only measurement-guided rule that beats Random at the 15% and 20% budgets and has the best median change at every budget. Frenet is effective only at the lightest budget, while NRS does not transfer from model selection to pruning. These results show that layer-wise structure provides signal for both interpretation and deployment decisions.

Problem

• layer-wise representation
• representation dynamics
• language models
• model analysis
• MTEB tasks

Innovation

• Layer-wise Representation Dynamics
• Grassmann geometry
• Neighborhood Retention Score
• Graph Filtration Mutual Information
• inference-time layer pruning