Stable-Drift: A Patient-Aware Latent Drift Replay Method for Stabilizing Representations in Continual Learning

2025-11-27

AI Summary

To address catastrophic forgetting in continual learning for medical imaging, this paper proposes a patient-level replay method grounded in latent representation drift. Unlike conventional replay strategies, our approach quantifies latent drift via multi-layer feature shifts across heterogeneous hospital data, yielding an interpretable and clinically traceable instability signal. Based on this signal, we construct a patient-aware memory buffer that selectively replays high-drift samples, balancing individual representativeness and population diversity. The method is compatible with both CNN and ViT backbones and incorporates naive domain adaptation to improve inter-hospital feature alignment. Evaluated on a cross-hospital COVID-19 CT classification benchmark, it significantly outperforms naive fine-tuning and random replay—demonstrating superior effectiveness, robustness under real-world clinical heterogeneity, and architectural generality across deep learning architectures.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

When deep learning models are sequentially trained on new data, they tend to abruptly lose performance on previously learned tasks, a critical failure known as catastrophic forgetting. This challenge severely limits the deployment of AI in medical imaging, where models must continually adapt to data from new hospitals without compromising established diagnostic knowledge. To address this, we introduce a latent drift-guided replay method that identifies and replays samples with high representational instability. Specifically, our method quantifies this instability via latent drift, the change in a sample internal feature representation after naive domain adaptation. To ensure diversity and clinical relevance, we aggregate drift at the patient level, our memory buffer stores the per patient slices exhibiting the greatest multi-layer representation shift. Evaluated on a cross-hospital COVID-19 CT classification task using state-of-the-art CNN and Vision Transformer backbones, our method substantially reduces forgetting compared to naive fine-tuning and random replay. This work highlights latent drift as a practical and interpretable replay signal for advancing robust continual learning in real world medical settings.

Problem

• Address catastrophic forgetting in continual learning for medical imaging
• Stabilize representations when adapting to new hospital data
• Use latent drift to guide replay for robust diagnostic knowledge

Innovation

• Latent drift quantifies representation instability after domain adaptation
• Patient-level aggregation ensures clinical relevance and sample diversity
• Drift-guided replay reduces catastrophic forgetting in medical imaging