Interactive Segmentation and Report Generation for CT Images

2025-03-05

AI Summary

Existing CT report generation methods suffer from poor interpretability and lack interactivity, hindering radiologists’ ability to dynamically refine lesion segmentation and reporting during image interpretation. To address this, we propose the first interactive framework for joint 3D lesion segmentation and structured report generation from volumetric CT scans. Our approach tightly integrates interactive prompts (points/bounding boxes), a 3D segmentation network, and a multi-attribute text generation module, enabling semantic alignment between segmentation masks and textual reports, as well as real-time co-updating. Radiologists can iteratively refine segmentations during diagnosis, with corresponding structured descriptions—including morphological attributes—generated synchronously. Evaluated on 15 lesion categories, our method achieves significant improvements in segmentation accuracy (+4.2% Dice) and clinical consistency of reports (+0.8/5 expert rating), enhancing diagnostic interpretability, controllability, and personalization. The code will be publicly released.

Technology Category

• Computer Vision: Segmentation
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Automated CT report generation plays a crucial role in improving diagnostic accuracy and clinical workflow efficiency. However, existing methods lack interpretability and impede patient-clinician understanding, while their static nature restricts radiologists from dynamically adjusting assessments during image review. Inspired by interactive segmentation techniques, we propose a novel interactive framework for 3D lesion morphology reporting that seamlessly generates segmentation masks with comprehensive attribute descriptions, enabling clinicians to generate detailed lesion profiles for enhanced diagnostic assessment. To our best knowledge, we are the first to integrate the interactive segmentation and structured reports in 3D CT medical images. Experimental results across 15 lesion types demonstrate the effectiveness of our approach in providing a more comprehensive and reliable reporting system for lesion segmentation and capturing. The source code will be made publicly available following paper acceptance.

Problem

• Improves diagnostic accuracy and workflow efficiency
• Enhances interpretability and patient-clinician understanding
• Enables dynamic adjustment of assessments during image review

Innovation

• Interactive 3D lesion morphology reporting framework
• Generates segmentation masks with attribute descriptions
• First integration of interactive segmentation in 3D CT