Tracking Any Point Methods for Markerless 3D Tissue Tracking in Endoscopic Stereo Images

2025-08-11

AI Summary

In minimally invasive laparoscopic surgery, severe tissue motion and limited field-of-view pose significant challenges for markerless 3D tracking. To address this, we propose an end-to-end, markerless 3D point tracking method leveraging stereo endoscopic video. Our approach innovatively integrates two CoTracker models: one for temporal point tracking across frames, and the other for stereo correspondence between left and right views—enabling seamless conversion from 2D track-any-point (TAP) outputs to high-accuracy 3D motion estimates. Evaluated on a chicken tissue phantom under realistic laparoscopic conditions, our method achieves a mean Euclidean distance error of 1.1 mm at tissue velocities up to 10 mm/s, outperforming existing state-of-the-art methods. Results demonstrate superior accuracy, robustness to motion and occlusion, and real-time capability (≥25 FPS), validating its clinical applicability in robot-assisted surgery. This work establishes a new paradigm for safe surgical navigation and context-aware intraoperative perception.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: State Estimation
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Security and Privacy: Tracking, profiling, and countermeasures against them
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Minimally invasive surgery presents challenges such as dynamic tissue motion and a limited field of view. Accurate tissue tracking has the potential to support surgical guidance, improve safety by helping avoid damage to sensitive structures, and enable context-aware robotic assistance during complex procedures. In this work, we propose a novel method for markerless 3D tissue tracking by leveraging 2D Tracking Any Point (TAP) networks. Our method combines two CoTracker models, one for temporal tracking and one for stereo matching, to estimate 3D motion from stereo endoscopic images. We evaluate the system using a clinical laparoscopic setup and a robotic arm simulating tissue motion, with experiments conducted on a synthetic 3D-printed phantom and a chicken tissue phantom. Tracking on the chicken tissue phantom yielded more reliable results, with Euclidean distance errors as low as 1.1 mm at a velocity of 10 mm/s. These findings highlight the potential of TAP-based models for accurate, markerless 3D tracking in challenging surgical scenarios.

Problem

• Accurate 3D tissue tracking in endoscopic stereo images
• Markerless tracking for minimally invasive surgery challenges
• Combining TAP networks for temporal and stereo tracking

Innovation

• Combines CoTracker models for 3D tracking
• Uses 2D TAP networks for markerless tracking
• Achieves 1.1 mm error in tissue tracking