Unifying Segment Anything in Microscopy with Multimodal Large Language Model

2025-05-16

AI Summary

Weak cross-domain generalization and low boundary precision plague biomedical microscopic image segmentation. To address this, we propose injecting vision-language knowledge (VLK) from multimodal large language models (MLLMs) into the Segment Anything Model (SAM), establishing a unified segmentation framework. We introduce two key innovations: (1) a Vision-Language Semantic Alignment (VLSA) module to enable cross-modal feature alignment, and (2) a Semantic Boundary Regularization (SBR) mechanism to enhance edge awareness—both preserving SAM’s zero-shot capability. Evaluated on nine in-domain datasets, our method improves Dice score and Segmentation Accuracy (SA) by 7.71% and 12.10%, respectively. On ten challenging cross-domain datasets, it still achieves gains of 6.79% in Dice and 10.08% in SA, setting a new state-of-the-art for microscopic image segmentation.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Accurate segmentation of regions of interest in biomedical images holds substantial value in image analysis. Although several foundation models for biomedical segmentation have currently achieved excellent performance on certain datasets, they typically demonstrate sub-optimal performance on unseen domain data. We owe the deficiency to lack of vision-language knowledge before segmentation. Multimodal Large Language Models (MLLMs) bring outstanding understanding and reasoning capabilities to multimodal tasks, which inspires us to leverage MLLMs to inject Vision-Language Knowledge (VLK), thereby enabling vision models to demonstrate superior generalization capabilities on cross-domain datasets. In this paper, we propose using MLLMs to guide SAM in learning microscopy crose-domain data, unifying Segment Anything in Microscopy, named uLLSAM. Specifically, we propose the Vision-Language Semantic Alignment (VLSA) module, which injects VLK into Segment Anything Model (SAM). We find that after SAM receives global VLK prompts, its performance improves significantly, but there are deficiencies in boundary contour perception. Therefore, we further propose Semantic Boundary Regularization (SBR) to prompt SAM. Our method achieves performance improvements of 7.71% in Dice and 12.10% in SA across 9 in-domain microscopy datasets, achieving state-of-the-art performance. Our method also demonstrates improvements of 6.79% in Dice and 10.08% in SA across 10 out-ofdomain datasets, exhibiting strong generalization capabilities. Code is available at https://github.com/ieellee/uLLSAM.

Problem

• Improving biomedical image segmentation accuracy across domains
• Enhancing vision models with vision-language knowledge for generalization
• Addressing boundary perception deficiencies in Segment Anything Model

Innovation

• Leveraging MLLMs for vision-language knowledge injection
• Introducing VLSA module for semantic alignment
• Using SBR to enhance boundary contour perception