Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Models

2026-07-15

AI Summary

Current medical multimodal large language models exhibit limited performance in 3D volumetric data understanding due to high annotation costs, opaque data practices, and a lack of explicit clinical reasoning. This work proposes a slice-level synthetic paradigm for constructing structured reasoning data that emulates radiologists’ diagnostic workflows by decomposing global clinical priors into fine-grained, layer-wise observations and generating interpretable chains of thought. The approach uniquely integrates clinical principles—such as sequential spatial tracking, multi-slice awareness, and differential exclusion—into the reasoning process, endowing standard 2D multimodal large language models with robust 3D spatial reasoning capabilities without requiring 3D pretraining. Experiments demonstrate that the proposed method significantly outperforms 2D baselines across multiple 3D medical benchmarks and achieves performance on par with computationally intensive native 3D architectures, effectively bridging the gap between efficiency and accuracy.

Abstract

While Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in 2D medical image understanding, their extension to 3D volumetric imaging remains hindered by prohibitive annotation costs and dataset opacity. Current data formats, predominantly consisting of rigid Visual Question Answering (VQA) pairs or unstructured final clinical reports, typically fail to capture explicit clinical reasoning. To address this limitation, we introduce a large-scale structured reasoning dataset constructed via a novel slice-wise data synthesis paradigm. Inspired by the genuine diagnostic workflow of radiologists, this paradigm models visual cognition by decomposing the complex 3D reading process, translating global clinical priors into fine-grained, per-slice observations that are subsequently synthesized into an interpretable Chain-of-Thought (CoT). Crucially, this synthesized reasoning framework enforces essential clinical principles: sequential spatial tracking, multi-slice spatial awareness for artifact mitigation, and differential exclusion. To validate this approach, we instruction-tune a standard 2D-pretrained MLLM baseline using the synthesized data to enhance its volumetric comprehension. Comprehensive evaluations across multiple 3D medical benchmarks demonstrate that our method yields significant performance improvements over the 2D baseline. Furthermore, the resulting model exhibits robust spatial reasoning capabilities and rivals resource-intensive native 3D architectures, effectively bridging the performance gap. Ultimately, this data-centric strategy unlocks deep volumetric understanding and highly interpretable clinical logic without requiring computationally expensive 3D-specific pre-training. The complete repository, including datasets and training workflows, is publicly available at https://github.com/2020420145009/hounsfield.

Problem

• 3D spatial reasoning
• medical multimodal large language models
• volumetric imaging
• clinical reasoning
• annotation cost

Innovation

• 3D spatial reasoning
• multimodal large language models
• structured reasoning dataset
• slice-wise synthesis
• Chain-of-Thought