The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imaging

2026-02-24

AI Summary

This work addresses the challenge of deploying medical imaging models when fine-tuned on local private data yet evaluated on unlabeled, out-of-distribution test batches from unseen clinical sites—a scenario where conventional mean ensembling fails due to domain shift. To overcome this, the authors propose an entropy-driven online model fusion method that dynamically constructs a batch-specific ensemble during forward propagation. By decoupling the encoder and classifier head and fusing them separately, the approach achieves efficient unsupervised domain adaptation without requiring backpropagation. Evaluated across nine medical and natural image classification benchmarks, the method consistently outperforms existing techniques under both standard and challenging conditions, while preserving the inference efficiency of a single model.

Technology Category

• Machine Learning: Ensemble Methods
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Model merging under unseen test-time distribution shifts often renders naive strategies, such as mean averaging unreliable. This challenge is especially acute in medical imaging, where models are fine-tuned locally at clinics on private data, producing domain-specific models that differ by scanner, protocol, and population. When deployed at an unseen clinical site, test cases arrive in unlabeled, non-i.i.d. batches, and the model must adapt immediately without labels. In this work, we introduce an entropy-adaptive, fully online model-merging method that yields a batch-specific merged model via only forward passes, effectively leveraging target information. We further demonstrate why mean merging is prone to failure and misaligned under heterogeneous domain shifts. Next, we mitigate encoder classifier mismatch by decoupling the encoder and classification head, merging with separate merging coefficients. We extensively evaluate our method with state-of-the-art baselines using two backbones across nine medical and natural-domain generalization image classification datasets, showing consistent gains across standard evaluation and challenging scenarios. These performance gains are achieved while retaining single-model inference at test-time, thereby demonstrating the effectiveness of our method.

Problem

• model merging
• domain shift
• medical imaging
• test-time adaptation
• distribution shift

Innovation

• entropy-adaptive
• model merging
• heterogeneous domain shifts
• online adaptation
• encoder-classifier decoupling