Ultrasound-CLIP: Semantic-Aware Contrastive Pre-training for Ultrasound Image-Text Understanding

2026-04-02

AI Summary

Existing vision-language pretraining models struggle to adapt to ultrasound imaging due to its high anatomical heterogeneity and complex diagnostic attributes. To address this, this work introduces US-365K, a large-scale ultrasound image–text dataset, along with UDT, a standardized ultrasound diagnostic taxonomy. The authors propose Ultrasound-CLIP, a semantic-aware contrastive learning framework that integrates semantic soft labels and a diagnosis-attribute-driven heterogeneous graph neural network to enable structured semantic reasoning. As the first image–text pretraining approach specifically designed for ultrasound, Ultrasound-CLIP achieves state-of-the-art performance on both classification and retrieval tasks, demonstrating exceptional generalization capabilities across zero-shot, linear probing, and fine-tuning evaluation settings.

Abstract

Ultrasound imaging is widely used in clinical diagnostics due to its real-time capability and radiation-free nature. However, existing vision-language pre-training models, such as CLIP, are primarily designed for other modalities, and are difficult to directly apply to ultrasound data, which exhibit heterogeneous anatomical structures and diverse diagnostic attributes. To bridge this gap, we construct US-365K, a large-scale ultrasound image-text dataset containing 365k paired samples across 52 anatomical categories. We establish Ultrasonographic Diagnostic Taxonomy (UDT) containing two hierarchical knowledge frameworks. Ultrasonographic Hierarchical Anatomical Taxonomy standardizes anatomical organization, and Ultrasonographic Diagnostic Attribute Framework formalizes nine diagnostic dimensions, including body system, organ, diagnosis, shape, margins, echogenicity, internal characteristics, posterior acoustic phenomena, and vascularity. Building upon these foundations, we propose Ultrasound-CLIP, a semantic-aware contrastive learning framework that introduces semantic soft labels and semantic loss to refine sample discrimination. Moreover, we construct a heterogeneous graph modality derived from UDAF's textual representations, enabling structured reasoning over lesion-attribute relations. Extensive experiments with patient-level data splitting demonstrate that our approach achieves state-of-the-art performance on classification and retrieval benchmarks, while also delivering strong generalization to zero-shot, linear probing, and fine-tuning tasks.

Problem

• ultrasound imaging
• vision-language pre-training
• image-text understanding
• anatomical heterogeneity
• diagnostic attributes

Innovation

• Ultrasound-CLIP
• vision-language pre-training
• semantic-aware contrastive learning
• ultrasonographic diagnostic taxonomy
• heterogeneous graph modality