Uncovering the Latent Potential of Deep Intermediate Representations

2026-05-21

AI Summary

Existing approaches typically rely solely on the final layer of deep models or employ simplistic fusion of shallow representations, overlooking the fact that task-relevant information is non-monotonically distributed across intermediate layers and cannot be reliably recovered through naive aggregation. This work systematically uncovers, for the first time, the distributional patterns of informative representations in intermediate layers and introduces a Layer-wise Optimal Embedding Selection (LOES) strategy grounded in spectral analysis, complemented by a geometric regularization loss (GeoReg). By leveraging constructive spectral methods along with orthogonality and isotropy constraints, the proposed framework precisely identifies task-discriminative subspaces and stabilizes their geometric structure. The method consistently outperforms baselines across diverse architectures, modalities, and data scales, exhibits performance gains with increasing model depth, and enables interpretable cross-lingual and cross-modal analyses.

Abstract

Foundational Models pretrained on huge amount of data learn representations that evolve across depth, forming a hierarchy of embeddings with distinct semantic content and geometric structure. Contrary to the widespread practice of using only the final layer or shallow mixtures, we show that task-relevant information is distributed non-monotonically across layers and cannot be recovered by naïve aggregation. Through a geometric and empirical study across multiple modalities, we show that effective transfer depends on identifying which layers encode task-discriminative structure and how their embeddings are geometrically organized. We introduce Layer-wise Optimal Embedding Selection (LOES), a constructive spectral method that identifies task-discriminative subspaces by minimizing residual error under orthogonality and isotropy constraints. To align fine-tuning with this selection principle, we further propose Geometric Regularization Loss (GeoReg), which enforces a simplicial structure on class manifolds and stabilizes representation geometry during fine-tuning. Across a wide range of architectures, depths, modalities, and data regimes, LOES consistently outperforms standard baselines, with gains that grow as model depth increases. Beyond accuracy, our method reveals how semantic factors are distributed across layers, thereby enabling cross-lingual and cross-modal interpretability analyses. Together, our results provide strong evidence that layerwise embedding geometry is not incidental but central to how deep models represent and transfer knowledge.

Problem

• intermediate representations
• layerwise geometry
• representation transfer
• task-discriminative structure
• foundation models

Innovation

• Layer-wise Embedding Selection
• Geometric Regularization
• Intermediate Representations
• Representation Geometry
• Task-discriminative Subspaces