A Minimal Model of Representation Collapse: Frustration, Stop-Gradient, and Dynamics

2026-04-11

AI Summary

Self-supervised representation learning often suffers from “representation collapse,” leading to a loss of discriminability. This work constructs a minimal embedding model under a classification-representation setting and, by integrating gradient flow dynamics with fixed-point analysis, reveals for the first time that a small number of “frustrated samples” can induce collapse through a slow-timescale mechanism. Theoretically, it is shown that perfectly separable data never collapse, whereas frustrated samples alone are sufficient to disrupt structural stability. Introducing stop-gradient effectively preserves finite inter-class separation and stabilizes non-collapsed solutions. This mechanism is validated in a linear teacher-student model, offering an analytically tractable theoretical framework for understanding and preventing representation collapse.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Self-supervised representation learning is central to modern machine learning because it extracts structured latent features from unlabeled data and enables robust transfer across tasks and domains. However, it can suffer from representation collapse, a widely observed failure mode in which embeddings lose discriminative structure and distinct inputs become indistinguishable. To understand the mechanisms that drive collapse and the ingredients that prevent it, we introduce a minimal embedding-only model whose gradient-flow dynamics and fixed points can be analyzed in closed form, using a classification-representation setting as a concrete playground where collapse is directly quantified through the contraction of label-embedding geometry. We illustrate that the model does not collapse when the data are perfectly classifiable, while a small fraction of frustrated samples that cannot be classified consistently induces collapse through an additional slow time scale that follows the early performance gain. Within the same framework, we examine collapse prevention by adding a shared projection head and applying stop-gradient at the level of the training dynamics. We analyze the resulting fixed points and develop a dynamical mean-field style self-consistency description, showing that stop-gradient enables non-collapsed solutions and stabilizes finite class separation under frustration. We further verify empirically that the same qualitative dynamics and collapse-prevention effects appear in a linear teacher-student model, indicating that the minimal theory captures features that persist beyond the pure embedding setting.

Problem

• representation collapse
• self-supervised learning
• frustration
• embedding dynamics
• discriminative structure

Innovation

• representation collapse
• stop-gradient
• frustration
• gradient-flow dynamics
• minimal model