InfoNCE Induces Gaussian Distribution

2026-02-27

AI Summary

This work uncovers the intrinsic mechanism by which the InfoNCE loss in contrastive learning induces representations to exhibit a Gaussian structure. Under the alignment and uniformity assumptions, it theoretically establishes—for the first time—that encoder outputs asymptotically follow a multivariate Gaussian distribution as the representation dimension increases. To relax these assumptions, the authors introduce a lightweight regularization term promoting low norm and high entropy. Leveraging probability limit theory, feature projection analysis, and extensive experiments across multiple architectures, they demonstrate the universality of this Gaussian phenomenon on both synthetic data and CIFAR-10. These findings provide an analytically tractable Gaussian representation model that advances the theoretical understanding of unsupervised contrastive learning.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Contrastive learning has become a cornerstone of modern representation learning, allowing training with massive unlabeled data for both task-specific and general (foundation) models. A prototypical loss in contrastive training is InfoNCE and its variants. In this work, we show that the InfoNCE objective induces Gaussian structure in representations that emerge from contrastive training. We establish this result in two complementary regimes. First, we show that under certain alignment and concentration assumptions, projections of the high-dimensional representation asymptotically approach a multivariate Gaussian distribution. Next, under less strict assumptions, we show that adding a small asymptotically vanishing regularization term that promotes low feature norm and high feature entropy leads to similar asymptotic results. We support our analysis with experiments on synthetic and CIFAR-10 datasets across multiple encoder architectures and sizes, demonstrating consistent Gaussian behavior. This perspective provides a principled explanation for commonly observed Gaussianity in contrastive representations. The resulting Gaussian model enables principled analytical treatment of learned representations and is expected to support a wide range of applications in contrastive learning.

Problem

• InfoNCE
• contrastive learning
• Gaussian distribution
• representation learning
• asymptotic analysis

Innovation

• InfoNCE
• contrastive learning
• Gaussian distribution
• representation learning
• asymptotic analysis