Segmentation, Detection and Explanation: A Unified Framework for CT Appearance Reasoning

2026-05-15

AI Summary

Existing CT image analysis methods are often confined to single tasks and lack the capacity for joint reasoning over anatomical structures and contextual information, limiting their ability to meet clinical demands for multi-granularity analysis. This work proposes a unified autoregressive vision-language framework that dynamically activates detection and segmentation heads via task-routing tokens and incorporates a “look closer” mechanism to enable coarse-to-fine region focusing. The model jointly generates visual outputs and interpretable textual descriptions in an end-to-end manner. It is the first large vision-language model to integrate task routing with a progressive look-closer strategy, unifying detection, segmentation, and appearance-based reasoning. The authors also introduce the first multimodal CT dataset annotated with pixel-level masks, bounding boxes, and structured textual descriptions. The method achieves Dice score improvements of 1.0% on BTCV and 1.7% on MosMed+, outperforming current state-of-the-art approaches.

Abstract

Recent progress in deep learning has significantly advanced CT image analysis, particularly for segmentation tasks. However, these advances are largely confined to image-level pattern recognition, with most methods lacking explicit anatomical or contextual reasoning. Large vision-language models introduce linguistic context into image analysis, yet most approaches typically focus on a single task, which is insufficient for clinical workflow analysis that requires multiple fine-grained types of analysis, such as anatomy detection and segmentation. In this paper, we propose a unified autoregressive framework that integrates language-guided visual reasoning into CT interpretation. Our method introduces task-routing tokens that trigger detection and segmentation heads conditioned on the hidden states of a large vision-language model, enabling coherent generation of visual outputs (e.g., masks and bounding boxes) and textual reasonings. To progressively enhance localisation accuracy and semantic clarity, we further design a"closer-look"mechanism that allows the model to perform progressive coarse-to-fine visits to regions of interest under refined fields of view. To support model training and evaluation, we curated a new multimodal CT dataset containing pixel-wise masks, bounding boxes, spatial prompts, and structured descriptions for visual objects constructed through an AI-assisted annotation process with human verification. Experiments on public benchmarks demonstrate consistent improvements over the SoTA, achieving up to 1.0% Dice on BTCV and 1.7% Dice on MosMed+, while additionally providing appearance reasoning outputs. The code and dataset will be available.

Problem

• CT image analysis
• anatomical reasoning
• multimodal segmentation
• detection
• clinical workflow

Innovation

• unified autoregressive framework
• task-routing tokens
• closer-look mechanism
• vision-language reasoning
• multimodal CT dataset