Understanding Self-Supervised Learning via Gaussian Mixture Models

2024-11-05

AI Summary

This paper investigates the theoretical foundations underlying the effectiveness of self-supervised learning (SSL) for dimensionality reduction in Gaussian Mixture Models (GMMs). Addressing non-isotropic GMMs, it models data augmentation as independent sampling from the same latent component, enabling a unified analysis of subspace discovery in contrastive (InfoNCE) and non-contrastive (SimSiam) SSL frameworks. The work provides the first rigorous proof that both paradigms exactly recover the Fisher-optimal dimension-reducing subspace—overcoming the isotropy assumption inherent in classical spectral methods. Furthermore, it reveals that multimodal contrastive learning (e.g., CLIP) possesses intrinsic noise robustness, automatically suppressing representation noise. The analysis integrates information theory, statistical learning theory, and spectral analysis, and is empirically validated on synthetic data, demonstrating strong robustness to non-isotropic structures.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Self-supervised learning attempts to learn representations from un-labeled data; it does so via a loss function that encourages the embedding of a point to be close to that of its augmentations. This simple idea performs remarkably well, yet it is not precisely theoretically understood why this is the case. In this paper we analyze self-supervised learning in a natural context: dimensionality reduction in Gaussian Mixture Models. Crucially, we define an augmentation of a data point as being another independent draw from the same underlying mixture component. We show that vanilla contrastive learning (specifically, the InfoNCE loss) is able to find the optimal lower-dimensional subspace even when the Gaussians are not isotropic -- something that vanilla spectral techniques cannot do. We also prove a similar result for"non-contrastive"self-supervised learning (i.e., SimSiam loss). We further extend our analyses to multi-modal contrastive learning algorithms (e.g., CLIP). In this setting we show that contrastive learning learns the subset of fisher-optimal subspace, effectively filtering out all the noise from the learnt representations. Finally, we corroborate our theoretical finding through synthetic data experiments.

Problem

• Analyzes self-supervised learning in Gaussian Mixture Models
• Explains contrastive learning's effectiveness in dimensionality reduction
• Extends analysis to multi-modal contrastive learning algorithms

Innovation

• Self-supervised learning via Gaussian Mixture Models
• InfoNCE loss for non-isotropic Gaussians
• Multi-modal contrastive learning filters noise