Unifying Self-Supervised Clustering and Energy-Based Models

2023-12-30

AI Summary

This work addresses fundamental failure modes in self-supervised learning (SSL)—notably clustering collapse and generative degeneration—by establishing a theoretical bridge between SSL and probabilistic generative modeling. Methodologically, we derive, from first principles, the implicit probabilistic structure underlying mainstream SSL objectives, and formulate a unified graphical model framework that jointly encompasses self-supervised clustering and energy-based modeling. We further propose a novel variational lower bound that enables joint optimization of discriminative and generative capabilities, integrating cluster alignment, energy modeling, and neural-symbolic integration. The bound is both theoretically sound and practically trainable. Empirically, our approach achieves state-of-the-art performance on SVHN, CIFAR-10, and CIFAR-100 across three key metrics: clustering quality, generation fidelity, and out-of-distribution detection accuracy. Moreover, it successfully solves symbol grounding—a challenging downstream task requiring semantic alignment between learned representations and symbolic concepts.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Visual Reasoning & Symbolic Representations
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Self-supervised learning excels at learning representations from large amounts of data. At the same time, generative models offer the complementary property of learning information about the underlying data generation process. In this study, we aim at establishing a principled connection between these two paradigms and highlight the benefits of their complementarity. In particular, we perform an analysis of self-supervised learning objectives, elucidating the underlying probabilistic graphical models and presenting a standardized methodology for their derivation from first principles. The analysis suggests a natural means of integrating self-supervised learning with likelihood-based generative models. We instantiate this concept within the realm of cluster-based self-supervised learning and energy models, introducing a lower bound proven to reliably penalize the most important failure modes. Our theoretical findings are substantiated through experiments on synthetic and real-world data, including SVHN, CIFAR10, and CIFAR100, demonstrating that our objective function allows to jointly train a backbone network in a discriminative and generative fashion, consequently outperforming existing self-supervised learning strategies in terms of clustering, generation and out-of-distribution detection performance by a wide margin. We also demonstrate that the solution can be integrated into a neuro-symbolic framework to tackle a simple yet non-trivial instantiation of the symbol grounding problem.

Problem

• Unify self-supervised learning and generative models
• Integrate clustering with energy-based generative models
• Improve clustering, generation, and out-of-distribution detection

Innovation

• Integrates self-supervised learning with generative models
• Introduces a lower bound for failure mode penalization
• Enables joint discriminative and generative network training