Probabilistic Variational Contrastive Learning

2025-06-11

AI Summary

Existing contrastive learning methods (e.g., SimCLR, SupCon) produce deterministic embeddings without uncertainty quantification. To address this, we propose VCL—the first decoder-free variational contrastive learning framework. VCL interprets the InfoNCE loss as a surrogate reconstruction term, imposes a uniform prior on the unit hypersphere, and models the posterior via a projected normal distribution to enable sampling-based probabilistic embeddings. We further introduce a normalized KL regularization term to effectively mitigate dimensional collapse. This work establishes the first rigorous variational inference foundation for contrastive learning. Experiments demonstrate that VCL matches or exceeds the classification accuracy of SimCLR and SupCon across multiple benchmarks, significantly increases mutual information between embeddings and class labels, and provides interpretable, sample-based uncertainty estimates for learned representations.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Deterministic embeddings learned by contrastive learning (CL) methods such as SimCLR and SupCon achieve state-of-the-art performance but lack a principled mechanism for uncertainty quantification. We propose Variational Contrastive Learning (VCL), a decoder-free framework that maximizes the evidence lower bound (ELBO) by interpreting the InfoNCE loss as a surrogate reconstruction term and adding a KL divergence regularizer to a uniform prior on the unit hypersphere. We model the approximate posterior $q_ heta(z|x)$ as a projected normal distribution, enabling the sampling of probabilistic embeddings. Our two instantiations--VSimCLR and VSupCon--replace deterministic embeddings with samples from $q_ heta(z|x)$ and incorporate a normalized KL term into the loss. Experiments on multiple benchmarks demonstrate that VCL mitigates dimensional collapse, enhances mutual information with class labels, and matches or outperforms deterministic baselines in classification accuracy, all the while providing meaningful uncertainty estimates through the posterior model. VCL thus equips contrastive learning with a probabilistic foundation, serving as a new basis for contrastive approaches.

Problem

• Lack of uncertainty quantification in contrastive learning embeddings
• Need for probabilistic embeddings in contrastive learning frameworks
• Dimensional collapse and mutual information enhancement in embeddings

Innovation

• VCL maximizes ELBO with InfoNCE and KL divergence
• Models posterior as projected normal distribution
• Replaces deterministic embeddings with probabilistic samples