From Pixels to Polygons: A Survey of Deep Learning Approaches for Medical Image-to-Mesh Reconstruction

2025-05-06

AI Summary

This paper addresses the critical computational medical problem of reconstructing 3D surface meshes from medical images. We systematically survey four mainstream deep learning–driven paradigms—template-based, statistical, generative, and implicit models—covering diverse anatomical structures (e.g., cardiac and neurological) and multimodal imaging modalities (CT, MRI). We propose, for the first time, a unified taxonomy that explicitly identifies three core challenges: topological correctness, geometric fidelity, and multimodal fusion. Our method integrates CNNs, GANs, VAEs, NeRFs, and differentiable rendering, and introduces a comprehensive evaluation framework incorporating Dice score, Hausdorff distance, curvature fidelity, and task-specific loss functions. We conduct cross-anatomical quantitative benchmarking, curate and standardize major public datasets, and establish the first clinical-simulation–oriented performance benchmark. This work provides systematic technical foundations for virtual clinical trials and mechanistic biomedical research.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multimodal Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

Deep learning-based medical image-to-mesh reconstruction has rapidly evolved, enabling the transformation of medical imaging data into three-dimensional mesh models that are critical in computational medicine and in silico trials for advancing our understanding of disease mechanisms, and diagnostic and therapeutic techniques in modern medicine. This survey systematically categorizes existing approaches into four main categories: template models, statistical models, generative models, and implicit models. Each category is analysed in detail, examining their methodological foundations, strengths, limitations, and applicability to different anatomical structures and imaging modalities. We provide an extensive evaluation of these methods across various anatomical applications, from cardiac imaging to neurological studies, supported by quantitative comparisons using standard metrics. Additionally, we compile and analyze major public datasets available for medical mesh reconstruction tasks and discuss commonly used evaluation metrics and loss functions. The survey identifies current challenges in the field, including requirements for topological correctness, geometric accuracy, and multi-modality integration. Finally, we present promising future research directions in this domain. This systematic review aims to serve as a comprehensive reference for researchers and practitioners in medical image analysis and computational medicine.

Problem

• Surveying deep learning methods for medical image-to-mesh reconstruction
• Analyzing four model categories for anatomical structure applications
• Identifying challenges in topological correctness and geometric accuracy

Innovation

• Deep learning transforms medical images to 3D meshes
• Four model categories: template, statistical, generative, implicit
• Evaluates methods across cardiac and neurological applications