Enhancing Free-hand 3D Photoacoustic and Ultrasound Reconstruction using Deep Learning

2025-02-05

AI Summary

To address the challenges of narrow field-of-view, inaccurate motion estimation, and poor multimodal structural visualization in handheld photoacoustic/ultrasound (PAUS) 3D reconstruction, this paper proposes MoGLo-Net—a sensor-free, end-to-end deep network. Methodologically, it introduces a novel global-local self-attention mechanism integrated with a patch-wise correlation volume to robustly model scanning motion. It also achieves, for the first time, joint 3D vascular reconstruction from B-mode, Doppler, and photoacoustic modalities. Quantitative and qualitative evaluations demonstrate significant improvements over state-of-the-art methods, enabling high-precision, markerless, real-time freehand 3D reconstruction. The source code is publicly available.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

This study introduces a motion-based learning network with a global-local self-attention module (MoGLo-Net) to enhance 3D reconstruction in handheld photoacoustic and ultrasound (PAUS) imaging. Standard PAUS imaging is often limited by a narrow field of view and the inability to effectively visualize complex 3D structures. The 3D freehand technique, which aligns sequential 2D images for 3D reconstruction, faces significant challenges in accurate motion estimation without relying on external positional sensors. MoGLo-Net addresses these limitations through an innovative adaptation of the self-attention mechanism, which effectively exploits the critical regions, such as fully-developed speckle area or high-echogenic tissue area within successive ultrasound images to accurately estimate motion parameters. This facilitates the extraction of intricate features from individual frames. Additionally, we designed a patch-wise correlation operation to generate a correlation volume that is highly correlated with the scanning motion. A custom loss function was also developed to ensure robust learning with minimized bias, leveraging the characteristics of the motion parameters. Experimental evaluations demonstrated that MoGLo-Net surpasses current state-of-the-art methods in both quantitative and qualitative performance metrics. Furthermore, we expanded the application of 3D reconstruction technology beyond simple B-mode ultrasound volumes to incorporate Doppler ultrasound and photoacoustic imaging, enabling 3D visualization of vasculature. The source code for this study is publicly available at: https://github.com/guhong3648/US3D

Problem

• Improves 3D reconstruction in handheld PAUS imaging
• Addresses motion estimation without external sensors
• Enables 3D visualization of complex vasculature

Innovation

• Motion-based learning network
• Global-local self-attention module
• Patch-wise correlation operation