Endo-4DGX: Robust Endoscopic Scene Reconstruction and Illumination Correction with Gaussian Splatting

2025-06-29

AI Summary

In endoscopic surgery, non-uniform illumination—such as low-light or overexposed regions—causes geometric distortion and optimization instability in 3D Gaussian Splatting (3DGS) reconstruction. To address this, we propose a lighting-adaptive 4D Gaussian rasterization framework. Our method introduces: (1) a lighting embedding mechanism that models local brightness variations at the Gaussian point level; (2) region-aware enhancement and spatially aware adjustment modules to achieve view-consistent brightness correction; and (3) the first integration of an exposure control loss into the 4DGS pipeline, enabling joint optimization of geometry and appearance. Evaluated on real endoscopic datasets, our approach significantly outperforms existing combinations of 3D reconstruction and image restoration methods, achieving state-of-the-art performance in both geometric accuracy and rendering quality. This provides a robust visual foundation for intraoperative 3D scene understanding and robotic navigation.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Accurate reconstruction of soft tissue is crucial for advancing automation in image-guided robotic surgery. The recent 3D Gaussian Splatting (3DGS) techniques and their variants, 4DGS, achieve high-quality renderings of dynamic surgical scenes in real-time. However, 3D-GS-based methods still struggle in scenarios with varying illumination, such as low light and over-exposure. Training 3D-GS in such extreme light conditions leads to severe optimization problems and devastating rendering quality. To address these challenges, we present Endo-4DGX, a novel reconstruction method with illumination-adaptive Gaussian Splatting designed specifically for endoscopic scenes with uneven lighting. By incorporating illumination embeddings, our method effectively models view-dependent brightness variations. We introduce a region-aware enhancement module to model the sub-area lightness at the Gaussian level and a spatial-aware adjustment module to learn the view-consistent brightness adjustment. With the illumination adaptive design, Endo-4DGX achieves superior rendering performance under both low-light and over-exposure conditions while maintaining geometric accuracy. Additionally, we employ an exposure control loss to restore the appearance from adverse exposure to the normal level for illumination-adaptive optimization. Experimental results demonstrate that Endo-4DGX significantly outperforms combinations of state-of-the-art reconstruction and restoration methods in challenging lighting environments, underscoring its potential to advance robot-assisted surgical applications. Our code is available at https://github.com/lastbasket/Endo-4DGX.

Problem

• Robust reconstruction of endoscopic scenes under uneven lighting
• Illumination correction for low-light and over-exposure conditions
• Enhancing rendering quality in dynamic surgical environments

Innovation

• Illumination-adaptive Gaussian Splatting for endoscopy
• Region-aware enhancement for sub-area lightness
• Spatial-aware adjustment for view-consistent brightness