NeuCME: Toward Dynamic Multimodal Continual Learning via Neural Combinatorics of Multiple Experts

2026-09-07

AI Summary

本文针对动态多模态持续学习中的时空灾难性遗忘和自适应多模态融合问题，提出NeuCME框架，通过专家组合机制有效学习和整合知识。

Abstract

Multimodal continual learning has recently shown great potential for developing agents with human-like intelligence by continuously learning new tasks across multiple modalities. However, existing methods typically assume that the set of modalities per task is predefined and fixed. In this paper, we investigate a more realistic learning setting, referred to as dynamic multimodal continual learning, in which the set of modalities may vary across tasks rather than remaining fixed. This setting involves two primary challenges: (i) spatio-temporal catastrophic forgetting and (ii) adaptive multimodal fusion. To address these challenges, we propose NeuCME (as shorthand for \textbf{Neu}ral \textbf{C}ombinatorics of \textbf{M}ultiple \textbf{E}xperts), a novel framework designed to effectively learn and integrate knowledge across tasks with varying modalities. The proposed NeuCME model comprises three key components, namely modality-combinational rehearsal, multi-gated mixture-of-experts, and task relevance-guided distillation. Furthermore, we formulate an evaluation metric to quantify the dynamism of task sequences and then set up a comprehensive benchmark with different degrees of dynamism. Extensive experiments using four real-world datasets demonstrate that the proposed NeuCME outperforms state-of-the-art methods markedly.

Problem

• Dynamic Multimodal Continual Learning
• Spatio-temporal Catastrophic Forgetting
• Adaptive Multimodal Fusion

Innovation

• dynamic multimodal continual learning
• spatio-temporal catastrophic forgetting
• adaptive multimodal fusion
• modality-combinational rehearsal
• multi-gated mixture-of-experts