🤖 AI Summary
To address the insufficient accuracy of monocular video-based depth estimation in endoscopic sinus surgery—particularly for sub-millimeter requirements (≤0.625 mm)—this paper proposes a novel NeRF-stereo collaborative iterative framework. It initializes coarse neural radiance field (NeRF) reconstruction, synthesizes virtual stereo views, and leverages differentiable stereo matching to derive self-supervised depth constraints, which are then back-propagated to refine the NeRF—enabling end-to-end metric-depth map generation. The method integrates neural radiance fields, differentiable rendering, virtual stereo synthesis, and multi-stage self-supervised optimization. On synthetic data, it achieves a depth error of 0.125 ± 0.443 mm. For real clinical endoscopic videos, the reconstructed surface exhibits a mean distance of only 0.269 mm from preoperative CT-derived ground-truth meshes—setting the state-of-the-art accuracy for monocular 3D reconstruction in endoscopic sinus surgery.
📝 Abstract
3D reconstruction in endoscopic sinus surgery (ESS) demands exceptional accuracy, with the mean error and standard deviation necessitating within the range of a single CT slice (0.625 mm), as the critical structures in the nasal cavity are situated within submillimeter distances from surgical instruments. This poses a formidable challenge when using conventional monocular endoscopes. Depth estimation is crucial for 3D reconstruction, yet existing depth estimation methodologies either suffer from inherent accuracy limitations or, in the case of learning-based approaches, perform poorly when applied to ESS despite succeeding on their original datasets. In this study, we present a novel, highly generalizable method that combines Neural Radiance Fields (NeRF) and stereo depth estimation for 3D reconstruction that can derive metric monocular depth. Our approach begins with an initial NeRF reconstruction yielding a coarse 3D scene, the subsequent creation of binocular pairs within coarse 3D scene, and generation of depth maps through stereo vision, These depth maps are used to supervise subsequent NeRF iteration, progressively refining NeRF and binocular depth, the refinement process continues until the depth maps converged. This recursive process generates high-accuracy depth maps from monocular endoscopic video. Evaluation in synthetic endoscopy shows a depth accuracy of 0.125 $pm$ 0.443 mm, well within the 0.625 mm threshold. Further clinical experiments with real endoscopic data demonstrate a mean distance to CT mesh of 0.269 mm, representing the highest accuracy among monocular 3D reconstruction methods in ESS.