Generalizing Supervised Contrastive learning: A Projection Perspective

2025-06-11

AI Summary

Supervised contrastive learning (SupCon) lacks a principled information-theoretic interpretation, particularly in terms of mutual information (MI), and suffers from rigid projection head design and suboptimal negative sample utilization. Method: We propose ProjNCE, a unified objective integrating self-supervised and supervised contrastive learning. First, we derive a novel contrastive loss grounded in a tight MI lower bound. Second, we introduce a learnable projection function that enables class-center-guided embedding optimization. Third, we incorporate a negative sample correction term to enhance discriminability. Contribution/Results: This work establishes the first theoretical connection between SupCon and MI, bridging a fundamental interpretability gap. Extensive experiments across multiple benchmark datasets demonstrate consistent improvements over both SupCon and cross-entropy baselines, validating both theoretical rigor and empirical efficacy.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Self-supervised contrastive learning (SSCL) has emerged as a powerful paradigm for representation learning and has been studied from multiple perspectives, including mutual information and geometric viewpoints. However, supervised contrastive (SupCon) approaches have received comparatively little attention in this context: for instance, while InfoNCE used in SSCL is known to form a lower bound on mutual information (MI), the relationship between SupCon and MI remains unexplored. To address this gap, we introduce ProjNCE, a generalization of the InfoNCE loss that unifies supervised and self-supervised contrastive objectives by incorporating projection functions and an adjustment term for negative pairs. We prove that ProjNCE constitutes a valid MI bound and affords greater flexibility in selecting projection strategies for class embeddings. Building on this flexibility, we further explore the centroid-based class embeddings in SupCon by exploring a variety of projection methods. Extensive experiments on multiple datasets and settings demonstrate that ProjNCE consistently outperforms both SupCon and standard cross-entropy training. Our work thus refines SupCon along two complementary perspective--mutual information interpretation and projection design--and offers broadly applicable improvements whenever SupCon serves as the foundational contrastive objective.

Problem

• Explores relationship between supervised contrastive learning and mutual information
• Introduces ProjNCE to unify supervised and self-supervised contrastive objectives
• Improves SupCon via mutual information interpretation and projection design

Innovation

• ProjNCE generalizes InfoNCE for supervised learning
• ProjNCE provides flexible projection strategies for embeddings
• ProjNCE consistently outperforms SupCon and cross-entropy