SCISSR: Scribble-Conditioned Interactive Surgical Segmentation and Refinement

2026-03-19

AI Summary

This work addresses the challenge of accurately segmenting tissues and instruments in surgical images, where irregular shapes, specular reflections, and occlusions necessitate costly pixel-level annotations and render conventional point- or box-based prompts insufficient for precise localization. To overcome this, the authors introduce freehand scribbles as a novel form of interactive prompt and propose a lightweight scribble encoder that converts sparse user-drawn scribbles into dense prompt embeddings. Integrated with spatially gated fusion and LoRA adapters, this approach enables efficient iterative refinement while keeping the SAM backbone frozen. Notably, it operates within existing prompt-driven frameworks without architectural modifications, ensuring strong transferability. On EndoVis 2018 and CholecSeg8k, the method achieves Dice scores of 95.41% and 96.30% in just five and three interaction rounds, respectively, substantially outperforming point-based prompting strategies.

Technology Category

• Computer Vision: Segmentation
• Natural Language Processing: Prompt Engineering / Prompting
• Humans and AI: Interaction Techniques and Devices

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Data transparency and provenance

Abstract

Accurate segmentation of tissues and instruments in surgical scenes is annotation-intensive due to irregular shapes, thin structures, specularities, and frequent occlusions. While SAM models support point, box, and mask prompts, points are often too sparse and boxes too coarse to localize such challenging targets. We present SCISSR, a scribble-promptable framework for interactive surgical scene segmentation. It introduces a lightweight Scribble Encoder that converts freehand scribbles into dense prompt embeddings compatible with the mask decoder, enabling iterative refinement for a target object by drawing corrective strokes on error regions. Because all added modules (the Scribble Encoder, Spatial Gated Fusion, and LoRA adapters) interact with the backbone only through its standard embedding interfaces, the framework is not tied to a single model: we build on SAM 2 in this work, yet the same components transfer to other prompt-driven segmentation architectures such as SAM 3 without structural modification. To preserve pre-trained capabilities, we train only these lightweight additions while keeping the remaining backbone frozen. Experiments on EndoVis 2018 demonstrate strong in-domain performance, while evaluation on the out-of-distribution CholecSeg8k further confirms robustness across surgical domains. SCISSR achieves 95.41% Dice on EndoVis 2018 with five interaction rounds and 96.30% Dice on CholecSeg8k with three interaction rounds, outperforming iterative point prompting on both benchmarks.

Problem

• surgical segmentation
• interactive refinement
• scribble prompting
• annotation efficiency
• occlusion handling

Innovation

• scribble prompting
• interactive segmentation
• surgical scene analysis
• promptable segmentation
• lightweight adaptation