Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Models

2026-07-05

AI Summary

Existing methods for evaluating transferability in 3D medical image segmentation rely on time-consuming fine-tuning, which struggles to meet the stringent demands for boundary precision and anatomical consistency. This work proposes the first fine-tuning-free, topology-driven framework that aligns sparse features with semantic labels via minimum spanning trees (MSTs). It assesses transferability at dual scales—local boundary token separability (LBTC) and global representation topological divergence (GRTD)—and incorporates a task-adaptive gated fusion mechanism. Theoretically, we prove that the MST leakage rate constitutes a finite-sample lower bound of the Bayes error and reveal that randomly initialized decoders stabilize topological alignment. Evaluated on a large-scale benchmark encompassing 114,000 3D medical images, our method achieves state-of-the-art performance, improving the weighted Kendall metric by 0.36 on average and accelerating evaluation by 56×.

Abstract

The growing number of medical vision foundation models highlights the need for effective model selection. However, mainstream selection methods rely on exhaustive fine-tuning, which is computationally expensive. Most of the existing Transferability Estimation (TE) metrics are primarily designed for image-level classification. They fail to preserve spatial relationships and fine-grained boundary details, which are crucial for the segmentation task. Additionally, while image-level tasks typically process a single feature vector per input, dense prediction tasks in 3D medical imaging require voxel-wise evaluation against dense annotations. To bridge these gaps, we propose a \textit{non-parametric, topology-driven} framework that estimates transferability directly from the alignment between the sparse 1-skeleton graph of dense features and semantic labels via Minimum Spanning Trees (MST). We decouple the alignment into two complementary geometric scales: Local Boundary-Aware Topological Consistency (LBTC) to assess boundary separability, where we prove that the MST leakage rate serves as a finite-sample lower bound on the Bayes error; and Global Representation Topology Divergence (GRTD) to evaluate the overall anatomical layout. Crucially, we formally justify a counterintuitive mechanism: Although without fine-tuning, the randomly initialized segmentation decoder acts as a topology-preserving spatial projector, reducing the variance of pairwise distance estimates and stabilizing global alignment evaluation. Fused via a task-adaptive gating mechanism, these dual metrics adapt to diverse clinical complexities. Evaluated on a large-scale benchmark of 114,000 3D medical volumes across diverse anatomical tasks, our topological framework achieves state-of-the-art transferability estimation with an average weighted Kendall (outperforming by 0.36) while accelerating evaluation by 56 times.

Problem

• Transferability Estimation
• 3D Medical Imaging
• Dense Prediction
• Segmentation
• Model Selection

Innovation

• topology-driven transferability estimation
• minimum spanning tree
• 3D medical segmentation
• non-parametric model selection
• dense prediction