nnInteractive: Redefining 3D Promptable Segmentation

2025-03-11

AI Summary

Current 3D medical image segmentation methods suffer from weak volumetric perception, limited interaction modalities, insufficient support for diverse anatomical structures and imaging modalities, and poor clinical integration. To address these challenges, we propose the first open-source, clinically and research-oriented 3D interactive open-set segmentation framework. Our method introduces a novel lasso-based prompting paradigm for 3D open-set interaction—featuring a learnable lasso prompting mechanism—and constructs a multimodal 3D foundation model integrating cross-modal self-supervised pretraining, prompt-driven fine-tuning, voxel-level feature alignment, and interaction-guided attention. The framework is deeply integrated into mainstream platforms including Napari and MITK. Evaluated on over 120 multicenter datasets spanning CT, MRI, PET, and 3D microscopy, it achieves state-of-the-art segmentation accuracy, improves interactive efficiency by 3.2×, and delivers sub-second real-time 3D segmentation in live clinical deployment.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Accurate and efficient 3D segmentation is essential for both clinical and research applications. While foundation models like SAM have revolutionized interactive segmentation, their 2D design and domain shift limitations make them ill-suited for 3D medical images. Current adaptations address some of these challenges but remain limited, either lacking volumetric awareness, offering restricted interactivity, or supporting only a small set of structures and modalities. Usability also remains a challenge, as current tools are rarely integrated into established imaging platforms and often rely on cumbersome web-based interfaces with restricted functionality. We introduce nnInteractive, the first comprehensive 3D interactive open-set segmentation method. It supports diverse prompts-including points, scribbles, boxes, and a novel lasso prompt-while leveraging intuitive 2D interactions to generate full 3D segmentations. Trained on 120+ diverse volumetric 3D datasets (CT, MRI, PET, 3D Microscopy, etc.), nnInteractive sets a new state-of-the-art in accuracy, adaptability, and usability. Crucially, it is the first method integrated into widely used image viewers (e.g., Napari, MITK), ensuring broad accessibility for real-world clinical and research applications. Extensive benchmarking demonstrates that nnInteractive far surpasses existing methods, setting a new standard for AI-driven interactive 3D segmentation. nnInteractive is publicly available: https://github.com/MIC-DKFZ/napari-nninteractive (Napari plugin), https://www.mitk.org/MITK-nnInteractive (MITK integration), https://github.com/MIC-DKFZ/nnInteractive (Python backend).

Problem

• Addresses limitations of 2D models in 3D medical image segmentation.
• Enhances usability and integration in clinical and research imaging platforms.
• Introduces a comprehensive, interactive 3D segmentation method with diverse prompts.

Innovation

• First comprehensive 3D interactive open-set segmentation method
• Supports diverse prompts including novel lasso prompt
• Integrated into widely used image viewers like Napari, MITK