Understanding Self-Supervised Learning via Latent Distribution Matching

2026-05-05

AI Summary

This work addresses the lack of a unified theoretical framework for self-supervised learning by formulating it as a latent variable distribution matching problem. The proposed approach achieves effective representation learning by simultaneously maximizing feature alignment and entropy in the latent space. This framework unifies contrastive, non-contrastive, and predictive methods under a common perspective and establishes identifiability of latent variables in nonlinear predictive settings. Building upon this foundation, the authors develop a nonlinear Bayesian filtering model that eliminates the need for negative sampling, integrating information-theoretic optimization with Kalman prediction principles. The resulting formulation provides a rigorous theoretical basis and general design principles for self-supervised learning.

Abstract

Self-supervised learning (SSL) excels at finding general-purpose latent representations from complex data, yet lacks a unifying theoretical framework that explains the diverse existing methods and guides the design of new ones. We cast SSL as latent distribution matching (LDM): learning representations that maximize their log-probability under an assumed latent model (alignment), while maximizing latent entropy to prevent collapse (uniformity). This view unifies independent component analysis with contrastive, non-contrastive, and predictive SSL methods, including stop gradient approaches. Leveraging LDM, we derive a nonlinear, sampling-free Bayesian filtering model with a Kalman-based predictor for high-dimensional timeseries. We further prove that predictive LDM yields identifiable latent representations under mild assumptions, even with nonlinear predictors. Overall, LDM clarifies the assumptions behind established SSL methods and provides principled guidance for developing new approaches.

Problem

• self-supervised learning
• theoretical framework
• latent representations
• unifying theory
• representation learning

Innovation

• Latent Distribution Matching
• Self-Supervised Learning
• Identifiable Representations
• Bayesian Filtering
• Nonlinear Predictors