Problem
Research questions and friction points this paper is trying to address.
Reconstructs deformable scenes from endoscopic videos
Overcomes limitations of known camera poses
Adapts to diverse camera and scene settings
BASED: Bundle-Adjusting Surgical Endoscopic Dynamic Video Reconstruction using Neural Radiance Fields
🤖 AI Summary
Dynamic 3D reconstruction of highly deformable endoscopic scenes under unknown camera poses remains challenging in minimally invasive surgery. Method: We propose the first neural radiance field (NeRF) method tailored for uncalibrated, strongly deforming surgical videos—departing from conventional NeRF assumptions of static scenes and fixed cameras. Our approach introduces an implicit dynamic NeRF framework that jointly optimizes camera extrinsics and non-rigid tissue deformation fields, integrated with bundle adjustment (BA) for end-to-end differentiable training. Results: Evaluated on multiple real robot-assisted laparoscopic surgery videos with varying lens parameters and complex tissue deformation patterns, our method achieves high-fidelity, complete, and robust real-time 3D reconstruction—outperforming existing state-of-the-art methods significantly.
📝 Abstract
Reconstruction of deformable scenes from endoscopic videos is important for many applications such as intraoperative navigation, surgical visual perception, and robotic surgery. It is a foundational requirement for realizing autonomous robotic interventions for minimally invasive surgery. However, previous approaches in this domain have been limited by their modular nature and are confined to specific camera and scene settings. Our work adopts the Neural Radiance Fields (NeRF) approach to learning 3D implicit representations of scenes that are both dynamic and deformable over time, and furthermore with unknown camera poses. We demonstrate this approach on endoscopic surgical scenes from robotic surgery. This work removes the constraints of known camera poses and overcomes the drawbacks of the state-of-the-art unstructured dynamic scene reconstruction technique, which relies on the static part of the scene for accurate reconstruction. Through several experimental datasets, we demonstrate the versatility of our proposed model to adapt to diverse camera and scene settings, and show its promise for both current and future robotic surgical systems.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Neural Radiance Fields for dynamic scene reconstruction
Unknown camera pose handling in endoscopic videos
Versatile adaptation to diverse surgical settings
🔎 Similar Papers
