CLA: Latent Alignment for Online Continual Self-Supervised Learning

2025-07-14

AI Summary

In online continual self-supervised learning (OCSL), representation forgetting arises due to streaming data, unknown task boundaries, and limited computational budgets. To address this, we propose Continual Latent-space Alignment (CLA), a framework that mitigates forgetting without replay, regularization, or explicit task identifiers. CLA employs a momentum encoder to maintain historical representations and implicitly aligns new and old latent features within each mini-batch via contrastive learning coupled with instantaneous online updates. Experimentally, CLA achieves faster convergence and superior accuracy over state-of-the-art online continual learning methods under identical computational constraints. Moreover, when used as a pretraining strategy, CLA yields significantly better downstream task performance than conventional i.i.d. pretraining. This work presents the first efficient online continual self-supervised representation learning method that operates without replay mechanisms or task boundary information.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Self-supervised learning (SSL) is able to build latent representations that generalize well to unseen data. However, only a few SSL techniques exist for the online CL setting, where data arrives in small minibatches, the model must comply with a fixed computational budget, and task boundaries are absent. We introduce Continual Latent Alignment (CLA), a novel SSL strategy for Online CL that aligns the representations learned by the current model with past representations to mitigate forgetting. We found that our CLA is able to speed up the convergence of the training process in the online scenario, outperforming state-of-the-art approaches under the same computational budget. Surprisingly, we also discovered that using CLA as a pretraining protocol in the early stages of pretraining leads to a better final performance when compared to a full i.i.d. pretraining.

Problem

• Addresses online continual self-supervised learning challenges
• Mitigates forgetting by aligning current and past representations
• Improves convergence and performance under fixed computational budget

Innovation

• Aligns current and past latent representations
• Speeds up online training convergence
• Improves final performance via pretraining