CADRE: Stable, Parameter Efficient Adaptation of Medical Vision Language Models with Bounded Forgetting and Prior Drift

2026-06-22

AI Summary

This work addresses the critical challenges of catastrophic forgetting and pretraining prior shift that medical vision-language models face when continuously learning new clinical imaging modalities, which can compromise clinical safety. To mitigate these issues, the authors propose the CADRE framework, which freezes the backbone network and integrates low-rank adaptation (LoRA) with a novel elastic weight consolidation mechanism. CADRE introduces a self-scaling, similarity-aware weight consolidation term and a prior-anchoring regularizer to effectively constrain embedding drift while preserving retained knowledge. The method exhibits scale invariance, eliminating the sensitivity to task order inherent in traditional elastic weight consolidation. Evaluated on three heterogeneous breast cancer imaging tasks, CADRE updates only 0.23% of parameters, reduces forgetting by nearly sevenfold (from 0.075 to 0.011, p=0.023), and achieves state-of-the-art performance in accuracy, SPQ, and forward/backward transfer.

Abstract

Medical vision-language models (VLMs) such as BiomedCLIP generalize broadly, but adapting them to a clinical service is as much a safety problem as an accuracy one. Updating a deployed model for a new imaging modality can fail silently in two ways that harm patients: it can forget modalities it already handled (catastrophic forgetting), and it can drift from its trustworthy pretrained prior toward modality-specific shortcuts. We study parameter-efficient continual adaptation through these two properties rather than leaderboard accuracy, presenting CADRE: a frozen-backbone framework combining low-rank adaptation (LoRA) with an online, self-scaling, similarity-aware elastic weight consolidation term that bounds retained-competence loss, and an anchor-to-prior penalty bounding embedding drift from the frozen prior. Two short guarantees, a bound on total consolidation mass and a scale-invariance property, remove the scale-related sources of vanilla EWC's order fragility. Using breast cancer across three maximally dissimilar modalities (histopathology, ultrasound, chest radiography) as a controlled cross-modality stress test, under a multi-seed, multi-order protocol with paired significance testing and training approximately 0.23% of parameters, CADRE attains the highest accuracy, SPQ, and backward transfer and the lowest forgetting among adapting methods, reducing forgetting roughly sevenfold versus the strongest regularized baseline (0.075 to 0.011; paired p=0.023) and achieving positive backward transfer where every baseline is negative. We frame these as stability properties aligned with clinical-safety desiderata, not a deployment guarantee; robustness to distribution shift and adversarial inputs is out of scope.

Problem

• catastrophic forgetting
• prior drift
• medical vision-language models
• continual adaptation
• clinical safety

Innovation

• continual learning
• catastrophic forgetting
• parameter-efficient adaptation
• elastic weight consolidation
• vision-language models