Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Models

2026-07-08

AI Summary

This work presents the first systematic survey of continual self-supervised learning (CSSL) in vision, addressing the challenge of enabling models to learn continuously from unlabeled data streams while mitigating catastrophic forgetting. By analyzing existing evaluation protocols, investigating the mechanisms through which self-supervised objectives confer robustness to forgetting, and integrating insights from loss landscape geometry and methodological taxonomies, the study establishes a unified classification framework encompassing six major anti-forgetting strategies—namely distillation, replay, regularization, and others. The survey clarifies the current state of CSSL research, highlights inconsistencies in evaluation practices, reveals a pathway toward large-scale continual pre-training, and identifies key challenges such as scalability and rapid adaptation.

Abstract

Traditionally, continual learning has assumed access to labeled data, yet many real-world applications -- such as lifelong robotics -- require models to adapt continuously from unlabeled streams. This has led to the development of continual self-supervised learning (CSSL), a rapidly growing area that lacks a dedicated, systematic review. In this work, we present a comprehensive survey of CSSL for vision, with connections to emerging vision-language settings. First, we analyze existing evaluation protocols and highlight inconsistencies that hinder fair comparison. We then examine why self-supervised objectives exhibit improved robustness to catastrophic forgetting, relating this to task-agnostic representations and smoother loss landscapes. Next, we organize existing methods into a unified taxonomy based on their forgetting-mitigation strategies, including distillation, replay, regularization, architectural approaches, model merging, and objective-level adaptation. Finally, we identify open challenges such as scalability and the need for fast adaptability. We argue that advancing CSSL requires moving beyond small-scale benchmarks towards continual pre-training paradigms for large-scale systems.

Problem

• continual learning
• self-supervised learning
• catastrophic forgetting
• vision models
• lifelong learning

Innovation

• Continual Self-Supervised Learning
• Catastrophic Forgetting
• Task-Agnostic Representations
• Unified Taxonomy
• Continual Pre-training