Single-Slice-to-3D Reconstruction in Medical Imaging and Natural Objects: A Comparative Benchmark with SAM 3D

2026-02-10

AI Summary

This study addresses the fundamental challenge of reliably reconstructing three-dimensional anatomical structures from a single two-dimensional medical image, which suffers from depth ambiguity and insufficient geometric priors. We present the first systematic evaluation of zero-shot single-slice 3D reconstruction performance across five state-of-the-art general-purpose 3D foundation models—SAM3D, Hunyuan3D-2.1, Direct3D, Hi3DGen, and TripoSG—on both medical and natural images. Using quantitative metrics based on voxel overlap and point cloud distance, we assess their fidelity in reconstructing anatomical and pathological structures. Results demonstrate that SAM3D achieves the highest topological similarity, while other models tend to oversimplify geometry. Critically, all methods exhibit limited reconstruction accuracy at the voxel level, underscoring the inherent limitations of single-view medical 3D reconstruction and highlighting the necessity of multi-view approaches.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Deep Neural Architectures and Foundation Models
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

A 3D understanding of anatomy is central to diagnosis and treatment planning, yet volumetric imaging remains costly with long wait times. Image-to-3D foundations models can solve this issue by reconstructing 3D data from 2D modalites. Current foundation models are trained on natural image distributions to reconstruct naturalistic objects from a single image by leveraging geometric priors across pixels. However, it is unclear whether these learned geometric priors transfer to medical data. In this study, we present a controlled zero-shot benchmark of single slice medical image-to-3D reconstruction across five state-of-the-art image-to-3D models: SAM3D, Hunyuan3D-2.1, Direct3D, Hi3DGen, and TripoSG. These are evaluated across six medical datasets spanning anatomical and pathological structures and two natrual datasets, using voxel based metrics and point cloud distance metrics. Across medical datasets, voxel based overlap remains moderate for all models, consistent with a depth reconstruction failure mode when inferring volume from a single slice. In contrast, global distance metrics show more separation between methods: SAM3D achieves the strongest overall topological similarity to ground truth medical 3D data, while alternative models are more prone to over-simplication of reconstruction. Our results quantify the limits of single-slice medical reconstruction and highlight depth ambiguity caused by the planar nature of 2D medical data, motivating multi-view image-to-3D reconstruction to enable reliable medical 3D inference.

Problem

• single-slice-to-3D
• medical imaging
• 3D reconstruction
• depth ambiguity
• image-to-3D

Innovation

• single-slice-to-3D
• medical image reconstruction
• zero-shot benchmark
• geometric priors
• depth ambiguity