Refining 3D Medical Segmentation with Verbal Instruction

2026-03-15

AI Summary

Existing 3D medical segmentation models often produce inaccurate shape predictions due to data scarcity, annotation noise, and distribution shifts, failing to meet clinical precision requirements. To address this, this work proposes CoWTalk—the first benchmark that pairs erroneous 3D arterial anatomies with natural language correction instructions—and introduces a language-driven, human-in-the-loop iterative refinement framework. Leveraging a set-based 3D shape representation, the method integrates multimodal understanding of textual instructions to progressively correct segmentation outputs. Experiments demonstrate that, even under severe initial errors, the proposed approach substantially improves reconstruction accuracy over existing baselines, validating the efficacy and practicality of language-guided refinement for clinical-grade 3D segmentation.

Technology Category

• Computer Vision: Segmentation
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Accurate 3D anatomical segmentation is essential for clinical diagnosis and surgical planning. However, automated models frequently generate suboptimal shape predictions due to factors such as limited and imbalanced training data, inadequate labeling quality, and distribution shifts between training and deployment settings. A natural solution is to iteratively refine the predicted shape based on the radiologists' verbal instructions. However, this is hindered by the scarcity of paired data that explicitly links erroneous shapes to corresponding corrective instructions. As an initial step toward addressing this limitation, we introduce CoWTalk, a benchmark comprising 3D arterial anatomies with controllable synthesized anatomical errors and their corresponding repairing instructions. Building on this benchmark, we further propose an iterative refinement model that represents 3D shapes as vector sets and interacts with textual instructions to progressively update the target shape. Experimental results demonstrate that our method achieves significant improvements over corrupted inputs and competitive baselines, highlighting the feasibility of language-driven clinician-in-the-loop refinement for 3D medical shapes modeling.

Problem

• 3D medical segmentation
• anatomical shape refinement
• verbal instruction
• clinician-in-the-loop
• data scarcity

Innovation

• language-driven refinement
• 3D medical segmentation
• clinician-in-the-loop
• vector set representation
• instruction-guided correction