Deep Learning-Based Semantic Segmentation for Real-Time Kidney Imaging and Measurements with Augmented Reality-Assisted Ultrasound

2025-06-30

AI Summary

Ultrasound (US) imaging of the kidneys suffers from non-standard imaging planes and highly dynamic operator-dependent acquisition, resulting in a steep learning curve and frequent visual attention shifts—compromising both efficiency and accuracy of renal volume quantification. To address this, we propose a tightly integrated deep learning (DL) and augmented reality (AR) framework for real-time analysis. Our dual-path AR-DL pipeline leverages nnU-Net, Segmenter, YOLO, and MedSAM/LiteMedSAM to enable real-time semantic segmentation of kidneys in US video streams and automated 3D volumetric reconstruction. Visualization is rendered with sub-50-ms latency on HoloLens 2, supporting Wi-Fi-based wireless streaming and multi-device interoperability. Evaluated on the Open Kidney Dataset, our method achieves high segmentation accuracy (Dice > 0.92) and real-time performance (>25 FPS), significantly reducing cognitive load while enhancing point-of-care diagnostics and medical education. The source code and complete pipeline are publicly released.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Hardware-aware ML
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Ultrasound (US) is widely accessible and radiation-free but has a steep learning curve due to its dynamic nature and non-standard imaging planes. Additionally, the constant need to shift focus between the US screen and the patient poses a challenge. To address these issues, we integrate deep learning (DL)-based semantic segmentation for real-time (RT) automated kidney volumetric measurements, which are essential for clinical assessment but are traditionally time-consuming and prone to fatigue. This automation allows clinicians to concentrate on image interpretation rather than manual measurements. Complementing DL, augmented reality (AR) enhances the usability of US by projecting the display directly into the clinician's field of view, improving ergonomics and reducing the cognitive load associated with screen-to-patient transitions. Two AR-DL-assisted US pipelines on HoloLens-2 are proposed: one streams directly via the application programming interface for a wireless setup, while the other supports any US device with video output for broader accessibility. We evaluate RT feasibility and accuracy using the Open Kidney Dataset and open-source segmentation models (nnU-Net, Segmenter, YOLO with MedSAM and LiteMedSAM). Our open-source GitHub pipeline includes model implementations, measurement algorithms, and a Wi-Fi-based streaming solution, enhancing US training and diagnostics, especially in point-of-care settings.

Problem

• Automates kidney volumetric measurements using deep learning
• Reduces cognitive load with AR-assisted ultrasound visualization
• Enhances real-time ultrasound usability for point-of-care diagnostics

Innovation

• Deep learning enables real-time kidney segmentation
• Augmented reality projects ultrasound display in view
• Open-source pipeline supports diverse ultrasound devices