ARport: An Augmented Reality System for Markerless Image-Guided Port Placement in Robotic Surgery

2026-02-15

AI Summary

This study addresses the disconnect between preoperative planning and intraoperative execution in robot-assisted surgery by proposing a markerless augmented reality system based on an optical see-through head-mounted display. The system reconstructs the surgical scene by fusing RGB-D and pose data, leverages foundation models for patient surface segmentation, and accurately overlays the preoperatively planned trocar layout onto real anatomical locations through surface registration. This work presents the first demonstration of fully external-sensor-free and markerless trocar visualization with minimal hardware requirements, significantly streamlining surgical setup. Experiments on human phantoms validate the spatial alignment between virtual planning and physical anatomy, thereby enhancing the system’s potential for clinical integration.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Computer Vision: Motion & Tracking

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Purpose: Precise port placement is a critical step in robot-assisted surgery, where port configuration influences both visual access to the operative field and instrument maneuverability. To bridge the gap between preoperative planning and intraoperative execution, we present ARport, an augmented reality (AR) system that automatically maps pre-planned trocar layouts onto the patient's body surface, providing intuitive spatial guidance during surgical preparation. Methods: ARport, implemented on an optical see-through head-mounted display (OST-HMD), operates without any external sensors or markers, simplifying setup and enhancing workflow integration. It reconstructs the operative scene from RGB, depth, and pose data captured by the OST-HMD, extracts the patient's body surface using a foundation model, and performs surface-based markerless registration to align preoperative anatomical models to the extracted patient's body surface, enabling in-situ visualization of planned trocar layouts. A demonstration video illustrating the overall workflow is available online. Results: In full-scale human-phantom experiments, ARport accurately overlaid pre-planned trocar sites onto the physical phantom, achieving consistent spatial correspondence between virtual plans and real anatomy. Conclusion: ARport provides a fully marker-free and hardware-minimal solution for visualizing preoperative trocar plans directly on the patient's body surface. The system facilitates efficient intraoperative setup and demonstrates potential for seamless integration into routine clinical workflows.

Problem

• port placement
• robotic surgery
• augmented reality
• markerless registration
• preoperative planning

Innovation

• markerless registration
• augmented reality
• robotic surgery
• trocar placement
• foundation model