MILE: Mixture of Incremental LoRA Experts for Continual Semantic Segmentation across Domains and Modalities

2026-05-05

AI Summary

This work addresses catastrophic forgetting in continual semantic segmentation under cross-domain and cross-modal task increments by proposing a modular, parameter-efficient continual learning framework. The approach freezes a pretrained backbone and instantiates lightweight LoRA experts for each new task, complemented by a prototype-guided gating mechanism that dynamically selects relevant experts. Coupled with a parameter isolation strategy, this design effectively balances model stability and plasticity. In contrast to existing methods, the framework substantially overcomes scalability limitations of expert-based models, enabling continual learning of numerous tasks with minimal additional parameters. Experimental results demonstrate state-of-the-art performance across multiple cross-domain and cross-modal continual segmentation benchmarks, while offering significant advantages in memory overhead and training efficiency.

Abstract

Continual semantic segmentation requires models to adapt to new domains or modalities without sacrificing performance on previously learned tasks. Expert-based learning, in which task-specific modules specialize in different domains, has proven effective in mitigating forgetting. These methods include dynamic expansion, which suffers from scalability issues, or parameter isolation, which constrains the ability to learn new tasks. We introduce Mixture of Incremental LoRA Experts (MILE), a modular and parameter-efficient framework for continual segmentation across both domains and modalities. MILE leverages Low-Rank Adaptation (LoRA) to instantiate lightweight experts for each new task while keeping the pretrained base network frozen. Each expert is trained exclusively on its task data, thus avoids overwriting previously learned information. A prototype-guided gating mechanism dynamically selects the most appropriate expert at inference. MILE achieves the benefits of expert-based learning while overcoming its scalability limitations. It requires only a marginal parameter increase per task and tens of LoRA adapters are needed before matching the size of a single full model, making it highly efficient in both training and storage. Across domain- and modality-incremental benchmarks, MILE achieves strong performance while ensuring better stability, plasticity, and scalability.

Problem

• continual semantic segmentation
• domain adaptation
• modality adaptation
• catastrophic forgetting
• lifelong learning

Innovation

• LoRA
• Continual Learning
• Semantic Segmentation
• Mixture of Experts
• Parameter Efficiency