Training-free Test-time Improvement for Explainable Medical Image Classification

2025-06-22

AI Summary

In medical image cross-domain deployment, concept bottleneck models (CBMs) suffer from concept-level shifts induced by variations in imaging protocols and staining procedures; however, expert annotation of concept labels is prohibitively expensive, hindering conventional fine-tuning. To address this, we propose the first backpropagation-free, retraining-free test-time adaptation method for CBMs: given only four target-domain images per class and their image-level labels, it dynamically applies dual mechanisms—confusing concept masking and discriminative concept amplification—via concept activation map analysis, enabling real-time correction of concept activations during forward inference. Evaluated on dermoscopic and white blood cell image datasets, our method significantly improves cross-domain classification accuracy while preserving source-domain performance. It overcomes the high-cost concept annotation bottleneck in clinical settings, simultaneously ensuring interpretability, robustness, and practical deployability.

Technology Category

• Computer Vision: Medical and Biological Imaging
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

Deep learning-based medical image classification techniques are rapidly advancing in medical image analysis, making it crucial to develop accurate and trustworthy models that can be efficiently deployed across diverse clinical scenarios. Concept Bottleneck Models (CBMs), which first predict a set of explainable concepts from images and then perform classification based on these concepts, are increasingly being adopted for explainable medical image classification. However, the inherent explainability of CBMs introduces new challenges when deploying trained models to new environments. Variations in imaging protocols and staining methods may induce concept-level shifts, such as alterations in color distribution and scale. Furthermore, since CBM training requires explicit concept annotations, fine-tuning models solely with image-level labels could compromise concept prediction accuracy and faithfulness - a critical limitation given the high cost of acquiring expert-annotated concept labels in medical domains. To address these challenges, we propose a training-free confusion concept identification strategy. By leveraging minimal new data (e.g., 4 images per class) with only image-level labels, our approach enhances out-of-domain performance without sacrificing source domain accuracy through two key operations: masking misactivated confounding concepts and amplifying under-activated discriminative concepts. The efficacy of our method is validated on both skin and white blood cell images. Our code is available at: https://github.com/riverback/TF-TTI-XMed.

Problem

• Address concept-level shifts in medical image classification
• Improve out-of-domain performance without expert annotations
• Enhance model explainability without sacrificing accuracy

Innovation

• Training-free confusion concept identification strategy
• Masking misactivated confounding concepts
• Amplifying under-activated discriminative concepts