Incrementally Learning Multiple Diverse Data Domains via Multi-Source Dynamic Expansion Model

2025-01-15

AI Summary

Existing continual learning approaches for multi-source heterogeneous data suffer from severe catastrophic forgetting and inefficient knowledge transfer during cross-domain task incremental learning. To address these challenges, we propose the Multi-Source Dynamic Expansion Model (MSDEM), which introduces a novel dynamic expandable attention mechanism and a dynamic graph-weighted router to enable parameter-efficient cross-domain reuse and positive knowledge transfer. MSDEM integrates multiple pre-trained backbones, dynamic expert expansion, graph-structured routing, and continual learning–specific optimization strategies. Evaluated on multi-domain continual learning benchmarks, MSDEM significantly outperforms state-of-the-art methods: it accelerates convergence on new tasks by 23.6%, maintains an average accuracy of 98.4% on previously learned tasks, and—uniquely among existing frameworks—achieves strong generalization, high stability, and scalable extensibility simultaneously.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Multi-modal Vision
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Continual Learning seeks to develop a model capable of incrementally assimilating new information while retaining prior knowledge. However, current research predominantly addresses a straightforward learning context, wherein all data samples originate from a singular data domain. This paper shifts focus to a more complex and realistic learning environment, characterized by data samples sourced from multiple distinct domains. We tackle this intricate learning challenge by introducing a novel methodology, termed the Multi-Source Dynamic Expansion Model (MSDEM), which leverages various pre-trained models as backbones and progressively establishes new experts based on them to adapt to emerging tasks. Additionally, we propose an innovative dynamic expandable attention mechanism designed to selectively harness knowledge from multiple backbones, thereby accelerating the new task learning. Moreover, we introduce a dynamic graph weight router that strategically reuses all previously acquired parameters and representations for new task learning, maximizing the positive knowledge transfer effect, which further improves generalization performance. We conduct a comprehensive series of experiments, and the empirical findings indicate that our proposed approach achieves state-of-the-art performance.

Problem

• Continuous Learning
• Knowledge Retention
• Complex Data

Innovation

• MSDEM
• Continuous Learning
• Multi-source Data