CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detection

2026-07-02

AI Summary

This work addresses the challenges of continual learning in multimodal large language models for anomaly detection, where task semantic entanglement and triple heterogeneity—across modalities, domains, and defect scales—hinder effective knowledge retention and transfer. To this end, the authors propose a parameter-efficient isolation-sharing collaborative fine-tuning framework. It introduces, for the first time in continual anomaly detection, a layer-adaptive mixture-of-experts architecture: task-specific experts are instantiated via PrivLoRA to enforce physical knowledge isolation, while layer-adaptive shared experts preserve cross-task representational consistency. A momentum-based mechanism dynamically updates critical shared parameters to facilitate knowledge transfer. Evaluated under class-incremental, cross-domain, and cross-modal continual learning settings, the method consistently outperforms existing approaches, demonstrating its effectiveness and robustness in complex, heterogeneous anomaly detection tasks.

Abstract

Multimodal Large Language Models (MLLMs) excel in diverse vision tasks, but full-parameter retraining is computationally expensive as real-world knowledge evolves. Existing continual learning methods often suffer from semantic entanglement in parameter spaces across tasks, impeding the continuous deployment of models. This challenge is especially pronounced in Anomaly Detection (AD), which exhibits triple heterogeneity across modalities, domains, and defect scale variability, significantly complicating multi-task knowledge transfer. In this paper, we propose CL-Anomaly, a parameter-efficient fine-tuning framework based on an isolation-sharing collaboration to enable continual learning for anomaly detection with MLLMs. We introduce the task-private expert PrivLoRA, which physically isolates task-specific subspaces in the parameter space to prevent semantic entanglement of anomaly knowledge in diverse scenarios. The Layer-Adaptive Shared Experts maintain cross-task representations within a unified feature space, enabling knowledge sharing between previous and new tasks. Furthermore, we propose a Layer-Adaptive Knowledge Transfer strategy that automatically selects and dynamically updates the layer-wise key shared experts of each task via a momentum-based mechanism, promoting effective knowledge transfer across related anomaly detection tasks. Extensive experiments across three continual learning scenarios for anomaly detection, including class-incremental, cross-domain, and cross-modal, demonstrate that CL-Anomaly outperforms state-of-the-art methods. Code is available at https://github.com/WenDongyp/CL-Anomaly.

Problem

• Continual Learning
• Anomaly Detection
• Multimodal Large Language Model
• Semantic Entanglement
• Parameter-Efficient Fine-Tuning

Innovation

• Continual Learning
• Anomaly Detection
• Mixture-of-Experts
• Parameter-Efficient Fine-Tuning
• Multimodal Large Language Models