Robustness of transferability estimation metrics for medical imaging

2026-08-07

AI Summary

This study addresses the limitations of existing transferability estimation metrics in medical image transfer learning, which are predominantly designed for natural images and struggle with class imbalance and instability across different experimental settings. For the first time, this work systematically evaluates the robustness of these metrics in medical imaging contexts by constructing miniature target datasets with varying sample sizes and multiple random seeds to isolate perturbations in target data. Through comprehensive comparisons involving diverse transferability estimation methods and classification evaluation metrics that account for class imbalance, the study reveals that minor variations in target data or choices of evaluation metrics can substantially alter the ranking of source models. Consequently, current transferability estimators exhibit consistently low agreement with actual performance rankings, highlighting their inadequacy in medical applications.

Abstract

In transfer learning, the choice of source model largely influences the performance on a target dataset. Still, selecting a fitting source remains a challenging task, especially in medical imaging where one has to decide between models pre-trained on off-the-shelf options, such as ImageNet, and domain specific datasets. Transferability estimation (TE) metrics address this problem by aiming to predict the best performing source model in a computationally cost effective way. However, previous work has reported conflicting TE metric performances due to differences in experimental setups. Moreover, most TE metrics are designed for and evaluated on natural images, while being optimized for accuracy, whereas in medical imaging metrics that are more robust to class imbalance are typically used. We study the impact of varying the target dataset as an isolated factor, by constructing miniature populations of different sample sizes and random seeds. In addition, we investigate the influence of the evaluation metric used to obtain the reference ranking. We find that small modifications to the target dataset change the rankings. Furthermore, we show that the choice of evaluation metric affects the reference rankings and therefore the evaluation of TE metrics. Overall, we observe a low agreement between rankings from TE metrics and reference. The code, model checkpoints and data splits used in this work are available through https://github.com/niclasclassen/robustness-of-transferability-estimation-metrics-for-medical-imaging.

Problem

• transferability estimation
• medical imaging
• transfer learning
• robustness
• evaluation metrics

Innovation

• transferability estimation
• medical imaging
• robustness
• class imbalance
• transfer learning