Information-Maximized Soft Variable Discretization for Self-Supervised Image Representation Learning

2025-01-07

AI Summary

To address high feature redundancy, poor interpretability, and overreliance on contrastive paradigms in unsupervised image representation learning, this paper proposes IMSVD—a self-supervised method that models the probability distribution of each latent dimension via soft variable discretization, directly optimizing representations through information-theoretic objectives. Key contributions include: (i) the first information-maximizing soft discretization mechanism, theoretically equivalent to contrastive learning yet operating in a non-contrastive framework; (ii) a joint cross-entropy loss that substantially reduces feature redundancy; and (iii) learned representations that are transformation-invariant, low-redundancy, and interpretable at the variable level. IMSVD achieves superior accuracy and inference efficiency over state-of-the-art methods across multiple downstream tasks. Moreover, its embedded features enable semantic attribution per latent dimension. The code is publicly available and supports cross-paradigm transfer.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Representation Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Self-supervised learning (SSL) has emerged as a crucial technique in image processing, encoding, and understanding, especially for developing today's vision foundation models that utilize large-scale datasets without annotations to enhance various downstream tasks. This study introduces a novel SSL approach, Information-Maximized Soft Variable Discretization (IMSVD), for image representation learning. Specifically, IMSVD softly discretizes each variable in the latent space, enabling the estimation of their probability distributions over training batches and allowing the learning process to be directly guided by information measures. Motivated by the MultiView assumption, we propose an information-theoretic objective function to learn transform-invariant, non-travail, and redundancy-minimized representation features. We then derive a joint-cross entropy loss function for self-supervised image representation learning, which theoretically enjoys superiority over the existing methods in reducing feature redundancy. Notably, our non-contrastive IMSVD method statistically performs contrastive learning. Extensive experimental results demonstrate the effectiveness of IMSVD on various downstream tasks in terms of both accuracy and efficiency. Thanks to our variable discretization, the embedding features optimized by IMSVD offer unique explainability at the variable level. IMSVD has the potential to be adapted to other learning paradigms. Our code is publicly available at https://github.com/niuchuangnn/IMSVD.

Problem

• Unsupervised Learning
• Image Recognition
• Large-scale Unlabeled Images

Innovation

• Information Maximization
• Soft Variable Discretization
• Image Learning Efficiency