Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learning

2025-05-20

AI Summary

To address catastrophic forgetting and degraded generalization in continual learning under sparse labeling, this paper proposes Task-Modulated Contrastive Learning (TMCL). TMCL is the first framework to integrate a top-down, predictive-coding-inspired modulation mechanism into contrastive learning, enabling dynamic construction of representation spaces that are simultaneously modulation-invariant and view-invariant over unsupervised streaming data. By decoupling affine modulation parameters, enforcing modulation-invariance constraints, and replaying historical modulation states, TMCL achieves fine-tuning-free class-incremental separation and representation stabilization. Under only 1% label availability, TMCL attains state-of-the-art performance on both class-incremental and transfer learning benchmarks, jointly overcoming the dual bottlenecks of label scarcity and catastrophic forgetting.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Biological brains learn continually from a stream of unlabeled data, while integrating specialized information from sparsely labeled examples without compromising their ability to generalize. Meanwhile, machine learning methods are susceptible to catastrophic forgetting in this natural learning setting, as supervised specialist fine-tuning degrades performance on the original task. We introduce task-modulated contrastive learning (TMCL), which takes inspiration from the biophysical machinery in the neocortex, using predictive coding principles to integrate top-down information continually and without supervision. We follow the idea that these principles build a view-invariant representation space, and that this can be implemented using a contrastive loss. Then, whenever labeled samples of a new class occur, new affine modulations are learned that improve separation of the new class from all others, without affecting feedforward weights. By co-opting the view-invariance learning mechanism, we then train feedforward weights to match the unmodulated representation of a data sample to its modulated counterparts. This introduces modulation invariance into the representation space, and, by also using past modulations, stabilizes it. Our experiments show improvements in both class-incremental and transfer learning over state-of-the-art unsupervised approaches, as well as over comparable supervised approaches, using as few as 1% of available labels. Taken together, our work suggests that top-down modulations play a crucial role in balancing stability and plasticity.

Problem

• Achieves continual learning with sparse supervision
• Prevents catastrophic forgetting in unsupervised settings
• Integrates top-down modulations for stability and plasticity

Innovation

• Task-modulated contrastive learning (TMCL) for continual learning
• Predictive coding principles integrate top-down information unsupervised
• Affine modulations improve class separation without weight changes