EWC-Guided Diffusion Replay for Exemplar-Free Continual Learning in Medical Imaging

2025-09-28

AI Summary

Medical imaging foundation models face significant challenges in continual learning: privacy regulations prohibit storing real patient data, while full retraining is computationally prohibitive. This paper proposes a sample-free continual learning framework that integrates class-conditional diffusion-based virtual sample replay with Elastic Weight Consolidation (EWC) to mitigate catastrophic forgetting without retaining original data. Our key contribution lies in uncovering the coupled effect of replay fidelity and Fisher-weighted parameter drift on forgetting, and introducing a lightweight ViT architecture for efficient, privacy-preserving incremental updates. Evaluated on the CheXpert multi-task continual learning benchmark, our method achieves an AUROC of 0.851—surpassing DER++’s forgetting rate by over 30% and approaching joint-training performance (0.869). Results demonstrate strong efficacy, robustness, and scalability, establishing a practical pathway for privacy-compliant continual adaptation of medical AI systems.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

Medical imaging foundation models must adapt over time, yet full retraining is often blocked by privacy constraints and cost. We present a continual learning framework that avoids storing patient exemplars by pairing class conditional diffusion replay with Elastic Weight Consolidation. Using a compact Vision Transformer backbone, we evaluate across eight MedMNIST v2 tasks and CheXpert. On CheXpert our approach attains 0.851 AUROC, reduces forgetting by more than 30% relative to DER exttt{++}, and approaches joint training at 0.869 AUROC, while remaining efficient and privacy preserving. Analyses connect forgetting to two measurable factors: fidelity of replay and Fisher weighted parameter drift, highlighting the complementary roles of replay diffusion and synaptic stability. The results indicate a practical route for scalable, privacy aware continual adaptation of clinical imaging models.

Problem

• Developing privacy-preserving continual learning for medical imaging models
• Reducing catastrophic forgetting without storing patient data exemplars
• Enabling efficient model adaptation while maintaining diagnostic performance

Innovation

• Uses class conditional diffusion replay
• Combines replay with Elastic Weight Consolidation
• Employs compact Vision Transformer backbone