An Empirical Analysis of Continual Learning for Heterogeneous Medical Visual Question Answering

2026-07-13

AI Summary

This study addresses the challenge of catastrophic forgetting in medical visual question answering (MedVQA) systems when continuously learning heterogeneous clinical tasks—such as classification, detection, and report generation—where balancing stability and plasticity remains difficult. The work presents the first systematic evaluation of multiple continual learning methods across five diverse MedVQA task sequences, integrating Low-Rank Adaptation (LoRA) fine-tuning to comprehensively analyze anti-forgetting capabilities, sensitivity to task ordering, and the evolution of adaptation parameters. Findings reveal that existing approaches exhibit unstable performance under interleaved heterogeneous task training, with task sequence significantly influencing forgetting severity. Moreover, characteristic patterns of parameter drift emerge across algorithms, elucidating how differences in task objectives and supervision modalities critically affect model stability.

Abstract

Deploying medical visual question answering (MedVQA) systems in real-world clinical settings requires models that adapt to new clinical tasks without forgetting previously acquired knowledge. Continual learning (CL) provides a practical framework for this setting. Despite rapid progress in medical vision-language models, the behavior of CL methods when training these models across heterogeneous MedVQA tasks remains underexplored. This work presents a systematic evaluation of CL for MedVQA across diverse clinical objectives, including classification, multi-label classification, detection, cell counting, and report generation. Specifically, we explore (1) the ability of existing CL methods to mitigate catastrophic forgetting; (2) their sensitivity to task ordering, analyzing how different task sequences influence performance retention and forgetting; and (3) the evolution of low-rank adaptation parameters as new tasks are learned, revealing patterns of weight drift under different CL methods. Our findings suggest that existing CL methods struggle to maintain stability-plasticity balance when tasks with different objectives and supervision formats are interleaved. Code and full experimental setup will be publicly available.

Problem

• Continual Learning
• Medical Visual Question Answering
• Catastrophic Forgetting
• Heterogeneous Tasks
• Stability-Plasticity Balance

Innovation

• Continual Learning
• Medical Visual Question Answering
• Catastrophic Forgetting
• Task Ordering
• Low-Rank Adaptation