TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Models

2026-07-10

AI Summary

Medical vision-language models often suffer performance degradation on out-of-distribution data due to domain shift and category bias. Existing few-shot adaptation methods exhibit instability and limited generalization under extremely low-data regimes. This work proposes a training-free few-shot adaptation approach that, during inference, leverages a small set of support samples to adaptively recalibrate class logits, enabling model-agnostic, efficient, and stable zero-shot generalization. By integrating class-level logit correction with cross-modal alignment, the method effectively mitigates domain shift and category confusion. It consistently outperforms trainable adaptation baselines across nine diverse medical imaging datasets encompassing X-ray, ultrasound, MRI, CT, and histopathology slides in out-of-distribution settings.

Abstract

Medical Vision-Language Models (VLMs) exhibit strong zero-shot performance, yet their effectiveness still declines on out-of-distribution (OOD) data due to domain shifts and class bias inherited from large-scale pretraining. Existing few-shot adaptation methods typically introduce additional trainable components, which can be unstable in extremely low-data regimes (e.g., 1-shot), and lack robustness on different medical data. We present TCLA, a purely training-free few-shot adaptation method for Medical VLMs, which is fast and model-agnostic. TCLA corrects inference logits based on a small set of support samples, boosting pretrained VLMs performance by improving inter-class deconfusion and reducing domain shift. Extensive experiments on nine datasets across multiple medical imaging modalities including X-ray, Ultrasound, MRI, CT, Histopathology, demonstrate that TCLA consistently improves OOD performance of Medical VLMs and, in most of cases, outperforms existing training-based adaptation methods.

Problem

• out-of-distribution
• domain shift
• class bias
• few-shot adaptation
• medical vision-language models

Innovation

• training-free adaptation
• medical vision-language models
• few-shot learning
• logit calibration
• out-of-distribution generalization