EndoLRMGS: Complete Endoscopic Scene Reconstruction combining Large Reconstruction Modelling and Gaussian Splatting

2025-03-28

AI Summary

In robotic-assisted surgery, endoscopic 3D reconstruction faces critical challenges including depth discontinuity noise, boundary ambiguity, and missing occluded surfaces. To address these, this paper proposes the first synergistic reconstruction framework integrating a Large Reconstruction Model (LRM) with Gaussian Splatting, augmented by an Orthogonal Perspective Joint Projection Optimization (OPjPO) strategy to enhance pose and scale estimation accuracy. The method jointly models deformable soft tissue and rigid surgical instruments, enabling complete, geometrically consistent dynamic scene reconstruction. Experiments demonstrate substantial improvements: 2D projection IoU for instrument regions increases by over 40%; PSNR rises from 3.82 to 11.07 for instruments; tissue rendering achieves PSNR = 49.87 (+49.41) and SSIM = 29.21 (+27.68), significantly outperforming state-of-the-art approaches.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: 3D Computer Vision
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Large-scale security measurements

Abstract

Complete reconstruction of surgical scenes is crucial for robot-assisted surgery (RAS). Deep depth estimation is promising but existing works struggle with depth discontinuities, resulting in noisy predictions at object boundaries and do not achieve complete reconstruction omitting occluded surfaces. To address these issues we propose EndoLRMGS, that combines Large Reconstruction Modelling (LRM) and Gaussian Splatting (GS), for complete surgical scene reconstruction. GS reconstructs deformable tissues and LRM generates 3D models for surgical tools while position and scale are subsequently optimized by introducing orthogonal perspective joint projection optimization (OPjPO) to enhance accuracy. In experiments on four surgical videos from three public datasets, our method improves the Intersection-over-union (IoU) of tool 3D models in 2D projections by>40%. Additionally, EndoLRMGS improves the PSNR of the tools projection from 3.82% to 11.07%. Tissue rendering quality also improves, with PSNR increasing from 0.46% to 49.87%, and SSIM from 1.53% to 29.21% across all test videos.

Problem

• Achieve complete endoscopic scene reconstruction in surgery
• Improve depth estimation accuracy at object boundaries
• Enhance 3D modeling of tools and tissue rendering

Innovation

• Combines LRM and GS for scene reconstruction
• Uses OPjPO to optimize position and scale
• Improves IoU and PSNR significantly