PR-ENDO: Physically Based Relightable Gaussian Splatting for Endoscopy

2024-11-19

arXiv.org

AI Summary

In colorectal endoscopic diagnosis, 3D reconstruction suffers from severe distortions and view-synthesis artifacts due to constrained camera trajectories and strong viewpoint-dependent lighting. To address this, we propose the first physically plausible relighting Gaussian splatting framework tailored for endoscopic imagery. Our method integrates 3D Gaussian splatting representations with a ray-marched diffuse MLP, physically based rendering (PBR), and coupled camera-light geometric modeling—enabling stable reconstruction and photorealistic relighting under limited rotational motion. Evaluated on both public and in-house endoscopic datasets, our approach significantly outperforms existing baselines: novel-view synthesis PSNR improves by 2.1–3.8 dB, while effectively suppressing lighting artifacts and viewpoint inconsistency. This work establishes a robust, interpretable paradigm for clinical endoscopic 3D visualization.

Technology Category

• Computer Vision: 3D Computer Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Endoscopic procedures are crucial for colorectal cancer diagnosis, and three-dimensional reconstruction of the environment for real-time novel-view synthesis can significantly enhance diagnosis. We present PR-ENDO, a framework that leverages 3D Gaussian Splatting within a physically based, relightable model tailored for the complex acquisition conditions in endoscopy, such as restricted camera rotations and strong view-dependent illumination. By exploiting the connection between the camera and light source, our approach introduces a relighting model to capture the intricate interactions between light and tissue using physically based rendering and MLP. Existing methods often produce artifacts and inconsistencies under these conditions, which PR-ENDO overcomes by incorporating a specialized diffuse MLP that utilizes light angles and normal vectors, achieving stable reconstructions even with limited training camera rotations. We benchmarked our framework using a publicly available dataset and a newly introduced dataset with wider camera rotations. Our methods demonstrated superior image quality compared to baseline approaches.

Problem

• Overcoming artifacts from constrained camera and lighting in endoscopy.
• Separating light effects from tissue properties for accurate 3D reconstruction.
• Enabling tissue modifications with physically accurate light responses.

Innovation

• Separates light effects from tissue properties
• Enhances 3D Gaussian Splatting with relightable model
• Uses specialized MLP for complex light effects