CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning

2026-06-30

AI Summary

This work addresses online continual self-supervised learning (OCSSL), which aims to learn stable representations from unlabeled data streams without explicit task boundaries and under strict memory constraints. The authors propose CLIMB, the first method to integrate structured replay buffering with regularization by introducing a centroid-based hierarchical memory mechanism that curates hard examples. CLIMB further combines knowledge distillation with contrastive learning to effectively mitigate representation drift while preserving coverage of diverse data distributions within a fixed memory budget. Evaluated on Split CIFAR-100, Split ImageNet-100, and a newly introduced protocol with irregular task distributions, CLIMB significantly outperforms existing OCSSL approaches.

Abstract

Online Continual Self-Supervised Learning (OCSSL) aims to learn representations from a continuous stream of unlabeled data, without knowledge of task boundaries and under memory constraints. Existing methods rely either on replay buffers that exploit latent space structure, or on regularization alone. We present CLIMB (Continual Learning with Intelligent Memory Bank), which combines both simultaneously. Our method introduces a hierarchical centroid-based memory, bounded in total number of stored images, combined with knowledge distillation on replayed examples to limit representation drift. The memory groups similar images into centroids, providing hard-to-discriminate examples for contrastive learning while covering the diversity of observed distributions. Experiments on Split CIFAR-100 and Split ImageNet-100, on standard benchmarks from the state-of-the-art as well as a new protocol with irregular task distributions show that CLIMB outperforms state-of-the-art OCSSL methods.

Problem

• Online Continual Learning
• Self-Supervised Learning
• Memory Constraints
• Unlabeled Data Stream
• Task Boundaries

Innovation

• centroid-based memory
• online continual learning
• self-supervised learning
• knowledge distillation
• contrastive learning