Source-Free Cross-Domain Continual Learning

2025-10-02

AI Summary

This work addresses cross-domain continual learning under privacy-sensitive settings where source-domain data is unavailable. We propose REFEREE, a source-free dynamic prompting collaboration framework that eliminates the need for source-data replay. Methodologically, it introduces a frequency-aware prompting collaboration mechanism, integrating uncertainty-weighted pseudo-label correction and low-frequency component enhancement to suppress label noise; employs kernelized linear discriminant analysis to freeze the backbone and mitigate catastrophic forgetting; and leverages large-scale vision-language models for collaborative inference. Its core innovation lies in modeling domain shift and task evolution without accessing any source-domain samples. Extensive experiments on multiple benchmarks demonstrate that REFEREE significantly outperforms existing source-dependent methods, achieving state-of-the-art continual learning performance under the source-free setting.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Computer Vision: Diffusion Models for Vision
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Although existing cross-domain continual learning approaches successfully address many streaming tasks having domain shifts, they call for a fully labeled source domain hindering their feasibility in the privacy constrained environments. This paper goes one step ahead with the problem of source-free cross-domain continual learning where the use of source-domain samples are completely prohibited. We propose the idea of rehearsal-free frequency-aware dynamic prompt collaborations (REFEREE) to cope with the absence of labeled source-domain samples in realm of cross-domain continual learning. REFEREE is built upon a synergy between a source-pre-trained model and a large-scale vision-language model, thus overcoming the problem of sub-optimal generalizations when relying only on a source pre-trained model. The domain shift problem between the source domain and the target domain is handled by a frequency-aware prompting technique encouraging low-frequency components while suppressing high-frequency components. This strategy generates frequency-aware augmented samples, robust against noisy pseudo labels. The noisy pseudo-label problem is further addressed with the uncertainty-aware weighting strategy where the mean and covariance matrix are weighted by prediction uncertainties, thus mitigating the adverse effects of the noisy pseudo label. Besides, the issue of catastrophic forgetting (CF) is overcome by kernel linear discriminant analysis (KLDA) where the backbone network is frozen while the classification is performed using the linear discriminant analysis approach guided by the random kernel method. Our rigorous numerical studies confirm the advantage of our approach where it beats prior arts having access to source domain samples with significant margins.

Problem

• Addresses source-free cross-domain continual learning without labeled source data
• Mitigates domain shifts using frequency-aware prompting and uncertainty weighting
• Overcomes catastrophic forgetting via kernel linear discriminant analysis

Innovation

• Frequency-aware prompting handles domain shifts
• Uncertainty-aware weighting mitigates noisy pseudo-labels
• Kernel linear discriminant analysis prevents catastrophic forgetting