Opportunistic Promptable Segmentation: Leveraging Routine Radiological Annotations to Guide 3D CT Lesion Segmentation

2026-01-30

AI Summary

This work addresses the challenge of 3D lesion segmentation, which typically relies on costly, pixel-level annotations, by leveraging the abundant yet underutilized coarse annotations—such as arrows and lines—routinely present in clinical PACS systems. The authors propose a novel paradigm termed "opportunistic promptable segmentation" and introduce SAM2CT, a model extending the SAM2 architecture with a prompt encoder that accepts arrow and line inputs, alongside a memory-conditioned mechanism tailored for 3D medical imaging. Evaluated on public datasets, SAM2CT achieves Dice scores of 0.649 and 0.757 using arrows and lines as prompts, respectively. In a clinical evaluation on 60 GSPS cases, 87% of the segmentations were rated by radiologists as acceptable or requiring only minor adjustments, demonstrating strong zero-shot generalization capabilities.

Technology Category

• Computer Vision: Segmentation
• Search and Optimization: Sampling/Simulation-based Search
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

The development of machine learning models for CT imaging depends on the availability of large, high-quality, and diverse annotated datasets. Although large volumes of CT images and reports are readily available in clinical picture archiving and communication systems (PACS), 3D segmentations of critical findings are costly to obtain, typically requiring extensive manual annotation by radiologists. On the other hand, it is common for radiologists to provide limited annotations of findings during routine reads, such as line measurements and arrows, that are often stored in PACS as GSPS objects. We posit that these sparse annotations can be extracted along with CT volumes and converted into 3D segmentations using promptable segmentation models, a paradigm we term Opportunistic Promptable Segmentation. To enable this paradigm, we propose SAM2CT, the first promptable segmentation model designed to convert radiologist annotations into 3D segmentations in CT volumes. SAM2CT builds upon SAM2 by extending the prompt encoder to support arrow and line inputs and by introducing Memory-Conditioned Memories (MCM), a memory encoding strategy tailored to 3D medical volumes. On public lesion segmentation benchmarks, SAM2CT outperforms existing promptable segmentation models and similarly trained baselines, achieving Dice similarity coefficients of 0.649 for arrow prompts and 0.757 for line prompts. Applying the model to pre-existing GSPS annotations from a clinical PACS (N = 60), SAM2CT generates 3D segmentations that are clinically acceptable or require only minor adjustments in 87% of cases, as scored by radiologists. Additionally, SAM2CT demonstrates strong zero-shot performance on select Emergency Department findings. These results suggest that large-scale mining of historical GSPS annotations represents a promising and scalable approach for generating 3D CT segmentation datasets.

Problem

• 3D CT segmentation
• radiological annotations
• sparse annotations
• GSPS
• lesion segmentation

Innovation

• Opportunistic Promptable Segmentation
• SAM2CT
• Memory-Conditioned Memories
• GSPS annotations
• 3D CT segmentation