How Does Perfect Fitting Affect Representation Learning? On the Training Dynamics of Representations in Deep Neural Networks

2024-05-27

arXiv.org

AI Summary

This work investigates the representational dynamics of overparameterized deep neural networks throughout full training—including beyond the point of perfect interpolation—with a focus on how optimizers (SGD vs. Adam) and architectures (ViT vs. ResNet) shape inter-layer representation similarity and learning patterns. Using centered kernel alignment (CKA) and linear-probe decision-region similarity as complementary metrics, we systematically analyze training trajectories across diverse width and depth configurations. We identify, for the first time, a representation phase transition in ViT encoders precisely at the perfect-fitting epoch; furthermore, deeper layers exhibit markedly stronger representational evolution during the epoch-wise double-descent regime. Our analysis reveals pronounced layer-wise heterogeneity in representation dynamics, with evolutionary intensity governed by layer depth, network width, and architectural class. These findings provide novel empirical evidence and conceptual insight into the implicit optimization dynamics of overparameterized models, advancing our understanding of how structure, optimization, and capacity jointly govern representation formation.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Models for Web evolution

Abstract

In this paper, we elucidate how representations in deep neural networks (DNNs) evolve during training. We focus on overparameterized learning settings where the training continues much after the trained DNN starts to perfectly fit its training data. We examine the evolution of learned representations along the entire training process, including its perfect fitting regime, and with respect to the epoch-wise double descent phenomenon. We explore the representational similarity of DNN layers, each layer with respect to its own representations throughout the training process. For this, we use two similarity metrics: (1) The centered kernel alignment (CKA) similarity; (2) Similarity of decision regions of linear classifier probes that we train for the DNN layers. Our extensive experiments discover training dynamics patterns that can emerge in layers depending on the relative layer-depth, DNN width, and architecture. We show that representations at the deeper layers evolve much more in the training when an epoch-wise double descent occurs. For Vision Transformer, we show that the perfect fitting threshold creates a transition in the evolution of representations across all the encoder blocks.

Problem

• Deep Neural Networks
• Optimization Algorithms
• Network Architectures

Innovation

• Overparameterization
• Learning Dynamics
• Vision Transformer vs ResNet