Toward Mask Annotation-Free Surgical Instrument Segmentation from Endoscopic Images Using Text-Prompted Segment Anything Model 3 (SAM3)

2026-08-09

AI Summary

This work addresses the limited scalability of existing surgical instrument segmentation methods, which typically rely on pixel-level annotations or manual prompts. The authors propose the first two-stage instance segmentation framework that operates without ground-truth masks or human interaction. In the first stage, SAM3 generates binary masks in a zero-shot manner using the generic text prompt “tool.” In the second stage, a fine-tuned vision-language model, Qwen, classifies the segmented regions into specific instrument categories. By avoiding direct use of instrument names as text prompts, the approach mitigates domain shift issues inherent in prior methods. Evaluated on the EndoVis 2017 and 2018 datasets, the proposed method significantly outperforms baseline approaches, demonstrating the feasibility and promise of annotation-free instance segmentation for surgical instruments.

Abstract

Surgical instrument segmentation is a fundamental task for computer-assisted interventions, yet most existing methods rely on pixel-level annotations or manual spatial prompts, which limit scalability and automation. The recently introduced Segment Anything Model 3 (SAM3) offers a pathway to annotation-free, automatic segmentation via text-based prompting; however, the instrument name as a text prompt could not be directly used due to a large domain gap. To overcome these limitations, we propose a two-stage framework that achieves instance-level segmentation without requiring ground truth masks or manual interaction. In the first stage, we leverage a natural-language-aligned generic prompt - "tool" - to produce binary masks using SAM3's zero-shot capability. In the second stage, these masks are extended to instance-level by integrating a vision-language model (Qwen) that is fine-tuned on SAM3-generated masked regions for instrument classification. We evaluate our approach on the EndoVis 2017 and 2018 datasets. Results show that, while our two-stage approach does not reach the performance of current fully supervised methods, it significantly outperforms the direct use of SAM3 for instance-level instrument segmentation with text prompts. Overall, our findings highlight both the limitations and potential of SAM3, suggesting a promising direction toward annotation-free surgical instrument segmentation.

Problem

• surgical instrument segmentation
• annotation-free
• text-prompted
• instance-level segmentation
• endoscopic images

Innovation

• annotation-free segmentation
• text-prompted SAM3
• surgical instrument segmentation
• vision-language model
• zero-shot segmentation