Surg - InvNeRF: Invertible NeRF for 3D tracking and reconstruction in surgical vision

2025-08-13

AI Summary

To address the longstanding challenges of long-term 3D point tracking, motion inconsistency, and 2D-only limitations in surgical scenes, this paper proposes InvNeRF—the first invertible neural radiance field framework tailored for test-time optimization (TTO). Methodologically, we introduce InvNeRF to TTO for the first time, integrating bidirectional deformation-canonical mapping, rendering-based supervision, and multi-scale HexPlanes representation; we further design an efficient pixel sampling strategy and a convergence criterion to ensure real-time performance and high-fidelity reconstruction. By incorporating kinematic priors, our approach achieves highly consistent joint 2D/3D tracking. On the STIR and SCARE benchmarks, our method improves average 2D tracking accuracy by nearly 50% over existing TTO approaches. Moreover, it is the first to enable robust 3D point tracking—significantly outperforming feedforward methods—and supports high-quality deformable surface reconstruction.

Technology Category

• Computer Vision: Motion & Tracking
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

We proposed a novel test-time optimisation (TTO) approach framed by a NeRF-based architecture for long-term 3D point tracking. Most current methods in point tracking struggle to obtain consistent motion or are limited to 2D motion. TTO approaches frame the solution for long-term tracking as optimising a function that aggregates correspondences from other specialised state-of-the-art methods. Unlike the state-of-the-art on TTO, we propose parametrising such a function with our new invertible Neural Radiance Field (InvNeRF) architecture to perform both 2D and 3D tracking in surgical scenarios. Our approach allows us to exploit the advantages of a rendering-based approach by supervising the reprojection of pixel correspondences. It adapts strategies from recent rendering-based methods to obtain a bidirectional deformable-canonical mapping, to efficiently handle a defined workspace, and to guide the rays' density. It also presents our multi-scale HexPlanes for fast inference and a new algorithm for efficient pixel sampling and convergence criteria. We present results in the STIR and SCARE datasets, for evaluating point tracking and testing the integration of kinematic data in our pipeline, respectively. In 2D point tracking, our approach surpasses the precision and accuracy of the TTO state-of-the-art methods by nearly 50% on average precision, while competing with other approaches. In 3D point tracking, this is the first TTO approach, surpassing feed-forward methods while incorporating the benefits of a deformable NeRF-based reconstruction.

Problem

• Long-term 3D point tracking in surgical scenarios
• Bidirectional deformable-canonical mapping for reconstruction
• Improving precision in 2D and 3D motion tracking

Innovation

• Invertible NeRF for 2D and 3D surgical tracking
• Multi-scale HexPlanes enable fast inference
• Bidirectional deformable-canonical mapping strategy