Adapting Vision-Language Foundation Model for Next Generation Medical Ultrasound Image Analysis

2025-06-10

AI Summary

In medical ultrasound image analysis, manual region-of-interest (ROI) delineation is time-consuming, highly subjective, and suffers from poor inter-observer reproducibility; meanwhile, existing vision-language foundation models exhibit limited performance due to substantial domain gaps between natural images and ultrasound data. To address these challenges, we propose the first ultrasound-specific vision-language foundation model adaptation framework. Our approach innovatively integrates a large language model as a text refiner, synergistically coupled with an ultrasound-domain-specific adapter and a multi-task head jointly optimized for segmentation and classification. Extensive experiments across six public ultrasound datasets demonstrate that our method consistently outperforms state-of-the-art vision-language and pure-vision baselines, achieving significant improvements in both segmentation accuracy and diagnostic classification performance. The source code is publicly available.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Medical ultrasonography is an essential imaging technique for examining superficial organs and tissues, including lymph nodes, breast, and thyroid. It employs high-frequency ultrasound waves to generate detailed images of the internal structures of the human body. However, manually contouring regions of interest in these images is a labor-intensive task that demands expertise and often results in inconsistent interpretations among individuals. Vision-language foundation models, which have excelled in various computer vision applications, present new opportunities for enhancing ultrasound image analysis. Yet, their performance is hindered by the significant differences between natural and medical imaging domains. This research seeks to overcome these challenges by developing domain adaptation methods for vision-language foundation models. In this study, we explore the fine-tuning pipeline for vision-language foundation models by utilizing large language model as text refiner with special-designed adaptation strategies and task-driven heads. Our approach has been extensively evaluated on six ultrasound datasets and two tasks: segmentation and classification. The experimental results show that our method can effectively improve the performance of vision-language foundation models for ultrasound image analysis, and outperform the existing state-of-the-art vision-language and pure foundation models. The source code of this study is available at href{https://github.com/jinggqu/NextGen-UIA}{GitHub}.

Problem

• Adapting vision-language models for medical ultrasound analysis
• Reducing manual labor in ultrasound image contouring
• Bridging domain gaps between natural and medical images

Innovation

• Domain adaptation for vision-language models
• Fine-tuning with large language model
• Task-driven heads for ultrasound analysis