SONAR: Self-Distilled Continual Pre-training for Domain Adaptive Audio Representation

2025-09-19

AI Summary

Existing audio representation models face dual challenges—high retraining costs and catastrophic forgetting—when adapting to continuously arriving unlabeled audio data. This paper proposes a continual pre-training framework for audio representation built upon the BEATs architecture, introducing a novel tripartite synergy mechanism: (1) joint sampling of new and old data; (2) contrastive regularization balanced via self-distillation; and (3) dynamically expandable tokenizer codebook. The framework jointly optimizes domain adaptation and general representation capability without compromising original task performance. Experiments across four heterogeneous audio domains demonstrate significant gains in new-domain performance (+8.2% average accuracy), while retaining prior-task accuracy with degradation under 0.5%, confirming strong resistance to catastrophic forgetting and superior cross-domain generalization.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Self-supervised learning (SSL) on large-scale datasets like AudioSet has become the dominant paradigm for audio representation learning. While the continuous influx of new, unlabeled audio presents an opportunity to enrich these static representations, a naive approach is to retrain the model from scratch using all available data. However, this method is computationally prohibitive and discards the valuable knowledge embedded in the previously trained model weights. To address this inefficiency, we propose SONAR (Self-distilled cONtinual pre-training for domain adaptive Audio Representation), a continual pre-training framework built upon BEATs. SONAR effectively adapts to new domains while mitigating catastrophic forgetting by tackling three key challenges: implementing a joint sampling strategy for new and prior data, applying regularization to balance specificity and generality, and dynamically expanding the tokenizer codebook for novel acoustic patterns. Experiments across four distinct domains demonstrate that our method achieves both high adaptability and robust resistance to forgetting.

Problem

• Adapting audio representations to new domains continually
• Mitigating catastrophic forgetting in continual pre-training
• Efficiently integrating new acoustic patterns without retraining

Innovation

• Self-distilled continual pre-training framework
• Joint sampling strategy for data
• Dynamic tokenizer expansion for acoustics