Self-Supervised Learning as Discrete Communication

2026-02-10

AI Summary

This work addresses the lack of structured semantic organization in continuous visual representations produced by existing self-supervised learning methods. It reformulates visual self-supervision as a discrete communication process between a teacher and a student network, where multi-label semantic information is transmitted through a fixed-capacity binary channel. The key innovations include using discrete binary codes as semantic carriers, introducing rate-distortion-inspired coding rate regularization to encourage compact and reusable representation structures, and combining periodic reinitialization of the projection head with element-wise binary cross-entropy loss. The proposed method substantially outperforms continuous alignment baselines across diverse tasks—including image classification, retrieval, dense prediction, and domain transfer—demonstrating that the learned binary codes constitute a compact yet semantically rich discrete visual language.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Most self-supervised learning (SSL) methods learn continuous visual representations by aligning different views of the same input, offering limited control over how information is structured across representation dimensions. In this work, we frame visual self-supervised learning as a discrete communication process between a teacher and a student network, where semantic information is transmitted through a fixed-capacity binary channel. Rather than aligning continuous features, the student predicts multi-label binary messages produced by the teacher. Discrete agreement is enforced through an element-wise binary cross-entropy objective, while a coding-rate regularization term encourages effective utilization of the constrained channel, promoting structured representations. We further show that periodically reinitializing the projection head strengthens this effect by encouraging embeddings that remain predictive across multiple discrete encodings. Extensive experiments demonstrate consistent improvements over continuous agreement baselines on image classification, retrieval, and dense visual prediction tasks, as well as under domain shift through self-supervised adaptation. Beyond backbone representations, we analyze the learned binary codes and show that they form a compact and informative discrete language, capturing semantic factors reusable across classes.

Problem

• Self-Supervised Learning
• Discrete Representation
• Visual Representation
• Structured Information
• Binary Communication

Innovation

• discrete communication
• self-supervised learning
• binary codes
• coding-rate regularization
• structured representations