Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learning

2024-10-01

arXiv.org

AI Summary

To address catastrophic forgetting—where pretrained knowledge is overwritten during domain incremental learning (DIL)—this paper proposes Duct, a dual-path knowledge consolidation framework. Duct jointly preserves historical domain knowledge at both the representation and classifier levels: first, via multi-stage backbone fusion and class-level semantic embedding mapping to achieve unified cross-domain feature modeling; second, through a novel semantic-guided mechanism for estimating legacy classifier weights, enabling joint alignment between the embedding space and classifier parameters. Evaluated on four standard DIL benchmarks, Duct significantly mitigates forgetting while enhancing cross-domain generalization, achieving state-of-the-art performance.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search
• Knowledge Representation and Reasoning: Knowledge Acquisition

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Domain-Incremental Learning (DIL) involves the progressive adaptation of a model to new concepts across different domains. While recent advances in pre-trained models provide a solid foundation for DIL, learning new concepts often results in the catastrophic forgetting of pre-trained knowledge. Specifically, sequential model updates can overwrite both the representation and the classifier with knowledge from the latest domain. Thus, it is crucial to develop a representation and corresponding classifier that accommodate all seen domains throughout the learning process. To this end, we propose DUal ConsolidaTion (Duct) to unify and consolidate historical knowledge at both the representation and classifier levels. By merging the backbone of different stages, we create a representation space suitable for multiple domains incrementally. The merged representation serves as a balanced intermediary that captures task-specific features from all seen domains. Additionally, to address the mismatch between consolidated embeddings and the classifier, we introduce an extra classifier consolidation process. Leveraging class-wise semantic information, we estimate the classifier weights of old domains within the latest embedding space. By merging historical and estimated classifiers, we align them with the consolidated embedding space, facilitating incremental classification. Extensive experimental results on four benchmark datasets demonstrate Duct's state-of-the-art performance. Code is available at https://github.com/Estrella-fugaz/CVPR25-Duct

Problem

• Prevent catastrophic forgetting in domain-incremental learning.
• Develop representation and classifier for multiple domains.
• Align historical and estimated classifiers with consolidated embeddings.

Innovation

• Dual consolidation for representation and classifier
• Merged backbone for multi-domain representation
• Classifier consolidation using semantic information