On the Importance of Embedding Norms in Self-Supervised Learning

2025-02-13

AI Summary

This work addresses the long-overlooked role of embedding norm in self-supervised learning (SSL). Motivated by the paradox that most SSL methods enforce unit-norm embeddings while discarding norm information, we theoretically establish—for the first time—that embedding norm directly governs SSL convergence rate and encodes sample anomaly (e.g., small norms indicate out-of-distribution or corrupted samples). Methodologically, we propose a norm-sensitivity diagnostic tool and a normalization-decoupled training framework, validated through theoretical analysis, controlled simulations, and large-scale experiments on SimCLR and BYOL. Our key contributions are: (i) uncovering a dual regulatory mechanism wherein norm jointly controls convergence speed and model confidence; (ii) achieving up to 2.3× training acceleration; and (iii) improving anomaly detection AUC by 12.7%. This work challenges the cosine-similarity-dominated paradigm in SSL and establishes a new theoretical foundation and practical benchmark for interpretability and robustness.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text

Abstract

Self-supervised learning (SSL) allows training data representations without a supervised signal and has become an important paradigm in machine learning. Most SSL methods employ the cosine similarity between embedding vectors and hence effectively embed data on a hypersphere. While this seemingly implies that embedding norms cannot play any role in SSL, a few recent works have suggested that embedding norms have properties related to network convergence and confidence. In this paper, we resolve this apparent contradiction and systematically establish the embedding norm's role in SSL training. Using theoretical analysis, simulations, and experiments, we show that embedding norms (i) govern SSL convergence rates and (ii) encode network confidence, with smaller norms corresponding to unexpected samples. Additionally, we show that manipulating embedding norms can have large effects on convergence speed. Our findings demonstrate that SSL embedding norms are integral to understanding and optimizing network behavior.

Problem

• Role of embedding norms in SSL
• Impact of norms on network convergence
• Norms encode network confidence levels

Innovation

• Embedding norms affect SSL convergence
• Smaller norms indicate unexpected samples
• Manipulating norms alters convergence speed