MedCAL-Bench: A Comprehensive Benchmark on Cold-Start Active Learning with Foundation Models for Medical Image Analysis

2025-08-05

AI Summary

Medical imaging cold-start active learning (CSAL) confronts dual challenges: scarce initial annotations and inadequate feature representation. Existing approaches rely on inefficient self-supervised feature extraction, while the potential of pretrained foundation models (FMs) remains systematically unexplored. This paper introduces the first FM-centric benchmark for CSAL, encompassing seven medical imaging datasets, fourteen FMs, and seven sampling strategies, supporting both classification and segmentation tasks. We conduct the first systematic evaluation of FMs’ feature transferability and sample selection efficacy in CSAL, uncovering critical model–task–strategy alignment patterns: DINO variants achieve optimal feature representation for segmentation; ALPS attains highest sampling efficiency in segmentation; and RepDiv excels in classification. The benchmark establishes a reproducible, empirically grounded evaluation framework for FM-driven efficient annotation in medical AI.

Technology Category

• Machine Learning: Active Learning
• Computer Vision: Medical and Biological Imaging
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Cold-Start Active Learning (CSAL) aims to select informative samples for annotation without prior knowledge, which is important for improving annotation efficiency and model performance under a limited annotation budget in medical image analysis. Most existing CSAL methods rely on Self-Supervised Learning (SSL) on the target dataset for feature extraction, which is inefficient and limited by insufficient feature representation. Recently, pre-trained Foundation Models (FMs) have shown powerful feature extraction ability with a potential for better CSAL. However, this paradigm has been rarely investigated, with a lack of benchmarks for comparison of FMs in CSAL tasks. To this end, we propose MedCAL-Bench, the first systematic FM-based CSAL benchmark for medical image analysis. We evaluate 14 FMs and 7 CSAL strategies across 7 datasets under different annotation budgets, covering classification and segmentation tasks from diverse medical modalities. It is also the first CSAL benchmark that evaluates both the feature extraction and sample selection stages. Our experimental results reveal that: 1) Most FMs are effective feature extractors for CSAL, with DINO family performing the best in segmentation; 2) The performance differences of these FMs are large in segmentation tasks, while small for classification; 3) Different sample selection strategies should be considered in CSAL on different datasets, with Active Learning by Processing Surprisal (ALPS) performing the best in segmentation while RepDiv leading for classification. The code is available at https://github.com/HiLab-git/MedCAL-Bench.

Problem

• Evaluating Foundation Models for cold-start active learning in medical imaging.
• Comparing feature extraction and sample selection strategies in CSAL.
• Assessing performance across diverse medical tasks and datasets.

Innovation

• First systematic FM-based CSAL benchmark
• Evaluates 14 FMs and 7 CSAL strategies
• Assesses feature extraction and sample selection