Preventing Latent Rehearsal Decay in Online Continual SSL with SOLAR

2026-04-12

AI Summary

This work addresses the issue of representation degradation and performance deterioration in online continual self-supervised learning, which often arises from overly stable replay mechanisms. The authors propose the “latent rehearsal decay” hypothesis and introduce two diagnostic metrics—Overlap and Deviation—to dynamically monitor changes in the latent space. Building on these insights, they design an adaptive buffer management strategy that leverages online proxy metrics to guide sample updates and incorporates an explicit Overlap loss function to modulate plasticity. Evaluated on standard OCSSL vision benchmarks, the proposed method achieves state-of-the-art performance, demonstrating superior convergence speed and final accuracy compared to existing approaches.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Computer Vision: Representation Learning for Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

This paper explores Online Continual Self-Supervised Learning (OCSSL), a scenario in which models learn from continuous streams of unlabeled, non-stationary data, where methods typically employ replay and fast convergence is a central desideratum. We find that OCSSL requires particular attention to the stability-plasticity trade-off: stable methods (e.g. replay with Reservoir sampling) are able to converge faster compared to plastic ones (e.g. FIFO buffer), but incur in performance drops under certain conditions. We explain this collapse phenomenon with the Latent Rehearsal Decay hypothesis, which attributes it to latent space degradation under excessive stability of replay. We introduce two metrics (Overlap and Deviation) that diagnose latent degradation and correlate with accuracy declines. Building on these insights, we propose SOLAR, which leverages efficient online proxies of Deviation to guide buffer management and incorporates an explicit Overlap loss, allowing SOLAR to adaptively managing plasticity. Experiments demonstrate that SOLAR achieves state-of-the-art performance on OCSSL vision benchmarks, with both high convergence speed and final performance.

Problem

• Online Continual Self-Supervised Learning
• Latent Rehearsal Decay
• stability-plasticity trade-off
• replay buffer
• latent space degradation

Innovation

• Latent Rehearsal Decay
• Online Continual Self-Supervised Learning
• Stability-Plasticity Trade-off
• Buffer Management
• Overlap Loss