Self-Supervised Learning Using Nonlinear Dependence

2025-01-31

AI Summary

In label-scarce settings, conventional self-supervised learning (SSL) methods overlook nonlinear dependencies among samples. To address this, we propose a unified representation learning framework that jointly models linear correlations and nonlinear dependencies. Specifically, we introduce the Hilbert–Schmidt Independence Criterion (HSIC) into SSL for the first time, leveraging reproducing kernel Hilbert spaces (RKHS) to capture sample-level and feature-level nonlinear dependencies. Our framework synergistically integrates contrastive learning with multi-view consistency optimization. This advances the theoretical foundations of SSL and substantially enhances representation discriminability. Extensive experiments on multiple benchmark datasets demonstrate consistent improvements in downstream task performance, with average accuracy gains of 2.3%–4.7% over strong baselines. These results validate both the effectiveness and generalizability of explicitly modeling nonlinear dependencies within SSL.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Self-supervised learning has gained significant attention in contemporary applications, particularly due to the scarcity of labeled data. While existing SSL methodologies primarily address feature variance and linear correlations, they often neglect the intricate relations between samples and the nonlinear dependencies inherent in complex data. In this paper, we introduce Correlation-Dependence Self-Supervised Learning (CDSSL), a novel framework that unifies and extends existing SSL paradigms by integrating both linear correlations and nonlinear dependencies, encapsulating sample-wise and feature-wise interactions. Our approach incorporates the Hilbert-Schmidt Independence Criterion (HSIC) to robustly capture nonlinear dependencies within a Reproducing Kernel Hilbert Space, enriching representation learning. Experimental evaluations on diverse benchmarks demonstrate the efficacy of CDSSL in improving representation quality.

Problem

• Self-supervised Learning
• Complex Data Relationships
• Sample Interrelations

Innovation

• CDSSL
• HSIC
• Self-Supervised Learning