Bootstrapping Vision-Language Model for Hysteroscopic Surgical Scene Segmentation

2026-08-10

AI Summary

This work addresses the challenges of semantic segmentation in hysteroscopic surgery, where lesion morphological similarity and artifacts such as specular reflections, motion blur, and fluid occlusion severely degrade performance. To tackle this, we propose VLM-hyster, the first vision–language model–based segmentation method for this domain. VLM-hyster integrates a pretrained image encoder with a Transformer decoder and incorporates class-specific textual prompts alongside a mask distillation mechanism to leverage textual semantics for enhanced attention to diagnostically critical regions and improved fine-grained class discrimination. Evaluated on a newly curated multicenter dataset of 4,020 high-resolution images, VLM-hyster significantly outperforms existing state-of-the-art methods. Its robustness and clinical applicability are further validated through assessments by gynecologists and prospective multicenter trials.

Abstract

Hysteroscopic surgical scene segmentation plays a pivotal role in understanding the hysteroscopic intraoperative environment as well as computer-assisted intervention. However, this task presents unique challenges due to the high morphological similarity among different lesions and the presence of artifacts such as specular reflections, motion blur, and fluid occlusions in surgical videos. In this work, we propose the first vision-language model (VLM)-based hysteroscopic surgical scene segmentation method, which performs pixel-wise localization for fifteen representative categories in hysteroscopic surgical scenes. Our VLM-hyster has a segmentation backbone that utilizes the pretrained image encoder for robust visual feature extraction, coupled with a transformer-based decoder for dense prediction. Moreover, we design category-specific text prompts and incorporate a masked distillation branch to filter out visual features with low correlation to the text prompts, enabling the model to focus more effectively on category-specific image regions and thereby enhancing segmentation performance. We collect a large multicentric hysteroscopic surgical scene dataset, containing 4,020 high-resolution images with detailed mask annotations, for model training and evaluation. Experimental results demonstrate that VLM-hyster substantially outperforms state-of-the-art AI models. Furthermore, extensive assessments by gynecologists, as well as multicentre and prospective validations, demonstrate VLM-hyster's robustness and generalizability. The results suggest that VLM-hyster earns considerable potential in enabling AI-assisted localization of surgical instruments and lesions in hysteroscopic surgeries. Code is available at https://github.com/viscom-tongji/VLM-hyster.

Problem

• hysteroscopic surgical scene segmentation
• morphological similarity
• surgical artifacts
• computer-assisted intervention
• intraoperative environment understanding

Innovation

• vision-language model
• surgical scene segmentation
• masked distillation
• category-specific prompts
• hysteroscopic surgery