DIMoE-Adapters: Dynamic Expert Evolution for Continual Learning in Vision-Language Models

2026-05-08

AI Summary

This work addresses the stability-plasticity imbalance and catastrophic forgetting in vision-language models during task-incremental learning across domains with significant distribution shifts. To this end, we propose a Dynamic Incremental Mixture-of-Experts Adapter framework, introducing two novel mechanisms: Self-Calibrated Expert Evolution (SCEE) and Prototype-Guided Expert Selection (PGES). These mechanisms dynamically construct and sparsely activate an expert pool, enabling efficient co-adaptation between old and new tasks. By overcoming the limitations of fixed architectures, our approach consistently outperforms state-of-the-art methods across diverse task-incremental settings, significantly enhancing the model’s continual learning capability and generalization performance.

Abstract

Continual learning enables vision-language models to accumulate knowledge and adapt to evolving tasks without retraining from scratch. However, in multi-domain task-incremental learning, large domain shifts intensify the stability-plasticity dilemma. Most existing methods rely on fixed architectures with statically allocated parameters, which limits adaptation to new domains and aggravates catastrophic forgetting. To address these challenges, we propose DIMoE-Adapters, a Dynamic Incremental Mixture-of-Experts Adapters framework that introduces a dynamic expert evolution paradigm to balance stability and plasticity. This paradigm is implemented through two collaborative components: Self-Calibrated Expert Evolution (SCEE) and Prototype-Guided Expert Selection (PGES). SCEE constructs and evolves a sparse expert pool through expert optimization dynamics, improving plasticity while reducing redundant capacity. PGES controls expert utilization based on the pool shaped by SCEE, improving stability across both previously encountered and unseen tasks. Extensive experiments show that DIMoE-Adapters outperforms previous state-of-the-art methods across various settings.

Problem

• continual learning
• vision-language models
• stability-plasticity dilemma
• catastrophic forgetting
• multi-domain task-incremental learning

Innovation

• Dynamic Expert Evolution
• Mixture-of-Experts Adapters
• Continual Learning
• Stability-Plasticity Trade-off
• Vision-Language Models