Real-time Rendering-based Surgical Instrument Tracking via Evolutionary Optimization

2026-03-11

AI Summary

This work addresses the challenge of accurate surgical instrument tracking in robotic surgery, where occlusions, complex joint structures, and degraded image quality hinder reliable pose estimation. The authors propose a novel tracking framework that integrates covariance matrix adaptation evolution strategy (CMA-ES) with batched GPU-based rendering. For the first time, CMA-ES is employed in rendering-based pose estimation to jointly optimize both the 6D pose of surgical instruments and their underlying joint configurations, eliminating the need for prior joint angle information and enabling support for bimanual instrument scenarios. By evaluating multiple pose hypotheses in parallel, the method achieves a favorable balance between real-time performance and robustness. Extensive experiments on both synthetic and real datasets demonstrate that the proposed approach significantly outperforms existing methods in terms of accuracy and speed, while exhibiting superior convergence stability and generalization capability.

Technology Category

• Intelligent Robots: State Estimation
• Computer Vision: Motion & Tracking
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Tracking, profiling, and countermeasures against them
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Accurate and efficient tracking of surgical instruments is fundamental for Robot-Assisted Minimally Invasive Surgery. Although vision-based robot pose estimation has enabled markerless calibration without tedious physical setups, reliable tool tracking for surgical robots still remains challenging due to partial visibility and specialized articulation design of surgical instruments. Previous works in the field are usually prone to unreliable feature detections under degraded visual quality and data scarcity, whereas rendering-based methods often struggle with computational costs and suboptimal convergence. In this work, we incorporate CMA-ES, an evolutionary optimization strategy, into a versatile tracking pipeline that jointly estimates surgical instrument pose and joint configurations. Using batch rendering to efficiently evaluate multiple pose candidates in parallel, the method significantly reduces inference time and improves convergence robustness. The proposed framework further generalizes to joint angle-free and bi-manual tracking settings, making it suitable for both vision feedback control and online surgery video calibration. Extensive experiments on synthetic and real-world datasets demonstrate that the proposed method significantly outperforms prior approaches in both accuracy and runtime.

Problem

• surgical instrument tracking
• real-time rendering
• partial visibility
• articulation design
• vision-based tracking

Innovation

• CMA-ES
• real-time rendering
• surgical instrument tracking
• evolutionary optimization
• batch rendering