Streamlining stereo differentiable rendering for marker-free real-time tracking of surgical robots

2026-07-14

AI Summary

This work addresses the challenge of marker-based tracking failure in surgical robots under occluded conditions by introducing the first markerless, real-time pose estimation method based on stereo differentiable rendering. The approach achieves robust dynamic tracking through inter-frame pose propagation and motion-adaptive hyperparameter optimization. A CUDA stream-parallel architecture is designed to jointly integrate segmentation and optimization pipelines, further accelerated by CUDA graphs for high-resolution image processing. Operating at 1080p@30fps, the system attains a static accuracy of 1.7 cm and 0.6°, with a mean dynamic 3D error of 1.2 cm (1.53 cm under occlusion). It outperforms FoundationPose by 11% in dynamic pose estimation and by 250% in static accuracy, while achieving a sixfold increase in inference speed.

Abstract

Purpose: Marker-based tracking of surgical robots is occlusion-prone in cluttered operating rooms. We evaluate stereo differentiable rendering for marker-free, real-time robot pose tracking, potentially improving safety, reducing setup time, and enabling multi-robot interaction. Methods: We extend the markerless pose estimation framework roboreg to online dynamic tracking via (i) sequential optimisation that propagates pose estimates across frames with motion-adaptive hyperparameter tuning, and (ii) CUDA stream parallelisation of segmentation and optimisation, combined with CUDA-graph accelerated segmentation. We evaluate on 38 unobstructed and 5 occluded displacement sequences with static start/end ground-truth calibrations and dynamic marker-based reference tracking. Results: We achieve real-time 1080p tracking at 30 fps (up from 14 fps for vanilla roboreg), matching the camera frame rate. Accuracy reaches 1.7 cm / 0.6 deg against static ground truth and 1.2 cm mean 3D error over 27,460 frames against the marker-based reference (1.53 cm over 1,242 occluded frames). Our method outperforms FoundationPose by 11% in dynamic estimation (63% under occlusion) and 250% in static estimation, with 6x faster inference. Conclusions: Stereo differentiable rendering enables real-time, high-resolution marker-free surgical robot tracking, on par with marker-based approaches and surpassing foundation-model baselines.

Problem

• marker-free tracking
• surgical robot
• occlusion
• real-time pose estimation
• stereo differentiable rendering

Innovation

• stereo differentiable rendering
• marker-free tracking
• real-time pose estimation
• CUDA stream parallelization
• surgical robot