Extend Your Horizon: A Device-Agnostic Surgical Tool Tracking Framework with Multi-View Optimization for Augmented Reality

2026-03-09

AI Summary

In surgical navigation, field-of-view occlusions frequently compromise conventional optical tracking, undermining the stability of augmented reality (AR) visualization. To address this limitation, this work proposes a device-agnostic multi-view optimization framework that integrates heterogeneous sensing modalities and constructs a dynamic scene graph to model sensor reliability in real time, thereby overcoming the line-of-sight dependency inherent in single-system approaches. By jointly leveraging multimodal fusion, dynamic scene graph representation, real-time reliability estimation, and a device-independent architecture, the method substantially enhances the robustness of instrument tracking and the consistency of AR visualization under occluded conditions.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Motion & Tracking
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health

Abstract

Surgical navigation provides real-time guidance by estimating the pose of patient anatomy and surgical instruments to visualize relevant intraoperative information. In conventional systems, instruments are typically tracked using fiducial markers and stationary optical tracking systems (OTS). Augmented reality (AR) has further enabled intuitive visualization and motivated tracking using sensors embedded in head-mounted displays (HMDs). However, most existing approaches rely on a clear line of sight, which is difficult to maintain in dynamic operating room environments due to frequent occlusions caused by equipment, surgical tools, and personnel. This work introduces a framework for tracking surgical instruments under occlusion by fusing multiple sensing modalities within a dynamic scene graph representation. The proposed approach integrates tracking systems with different accuracy levels and motion characteristics while estimating tracking reliability in real time. Experimental results demonstrate improved robustness and enhanced consistency of AR visualization in the presence of occlusions.

Problem

• surgical tool tracking
• occlusion
• augmented reality
• optical tracking
• surgical navigation

Innovation

• device-agnostic
• multi-view optimization
• surgical tool tracking
• occlusion robustness
• dynamic scene graph