Continual Model Merging with Test-Time Adaptation for Whole-Slide Image Analysis

2026-07-06

AI Summary

This work addresses catastrophic forgetting and distribution shift in whole-slide image (WSI) continual learning under the challenging no-replay setting. The authors propose a test-time adaptation (TTA)-guided model fusion approach: individual task models are independently fine-tuned during training, and at test time, fusion weights are dynamically adjusted using unlabeled target data to balance adaptation to new tasks with retention of previously acquired knowledge. This study presents the first systematic evaluation of such a strategy in continual histopathological image analysis, revealing its sensitivity to task order and underlying knowledge retention mechanisms. Experiments across six TCGA cancer subtype cohorts demonstrate that the method significantly improves performance and enhances historical knowledge preservation under both class-incremental (CLASS-IL) and task-incremental (TASK-IL) settings, though its efficacy is modulated by the interplay between current distribution adaptation and accumulated knowledge.

Abstract

Model merging offers a practical alternative to conventional continual learning by integrating independently fine-tuned models without retaining previous training data. Recent state-of-the-art model merging methods employ test-time adaptation (TTA-guided merging) to address distribution shifts by adjusting merging-related variables using unlabeled target data. However, these methods have primarily been studied in multi-task or single-target settings, and their behavior under sequential continual learning remains insufficiently understood. We present a benchmark study that maps this family of methods to rehearsal-free continual Whole Slide Image classification and evaluates them against traditional continual-learning approaches. Experiments on six TCGA cancer-subtyping cohorts cover CLASS-IL and TASK-IL scenarios, in-domain and out-of-domain evaluation, and different task orders. The results show that adapting model merging at test time can provide strong task-specific performance and improve retention of previously acquired knowledge without storing historical WSIs. Nevertheless, performance remains sensitive to task order and to the interaction between adaptation on the current distribution and accumulated knowledge. This benchmark identifies model merging with test-time adaptation as a promising direction for continual computational pathology and motivates future methods that balance adaptation to domain shift with explicit preservation of historical knowledge.

Problem

• continual learning
• model merging
• test-time adaptation
• whole-slide image
• distribution shift

Innovation

• model merging
• test-time adaptation
• continual learning
• whole-slide image
• computational pathology