MGSR: 2D/3D Mutual-boosted Gaussian Splatting for High-fidelity Surface Reconstruction under Various Light Conditions

2025-03-07

AI Summary

To address the longstanding trade-off between rendering fidelity and geometric accuracy in high-fidelity surface reconstruction, this paper proposes a dual-branch joint optimization framework unifying 2D and 3D Gaussian Splatting (GS). A geometry-guided reflectance/transmittance decomposition module enables illumination decoupling, while bidirectional supervision and an alternating training strategy—incorporating warm-start initialization and early stopping—enforce strong geometric consistency between rendering and reconstruction tasks. Crucially, this work achieves the first mutually reinforcing co-optimization of 2D-GS and 3D-GS, breaking the conventional performance bottleneck. Evaluated on multi-illumination synthetic and real-world datasets, our method improves PSNR by 1.8 dB and reduces Chamfer distance by 32% over state-of-the-art single-task approaches, demonstrating significant gains in both visual quality and geometric precision.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Mixed Discrete/Continuous Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Novel view synthesis (NVS) and surface reconstruction (SR) are essential tasks in 3D Gaussian Splatting (3D-GS). Despite recent progress, these tasks are often addressed independently, with GS-based rendering methods struggling under diverse light conditions and failing to produce accurate surfaces, while GS-based reconstruction methods frequently compromise rendering quality. This raises a central question: must rendering and reconstruction always involve a trade-off? To address this, we propose MGSR, a 2D/3D Mutual-boosted Gaussian splatting for Surface Reconstruction that enhances both rendering quality and 3D reconstruction accuracy. MGSR introduces two branches--one based on 2D-GS and the other on 3D-GS. The 2D-GS branch excels in surface reconstruction, providing precise geometry information to the 3D-GS branch. Leveraging this geometry, the 3D-GS branch employs a geometry-guided illumination decomposition module that captures reflected and transmitted components, enabling realistic rendering under varied light conditions. Using the transmitted component as supervision, the 2D-GS branch also achieves high-fidelity surface reconstruction. Throughout the optimization process, the 2D-GS and 3D-GS branches undergo alternating optimization, providing mutual supervision. Prior to this, each branch completes an independent warm-up phase, with an early stopping strategy implemented to reduce computational costs. We evaluate MGSR on a diverse set of synthetic and real-world datasets, at both object and scene levels, demonstrating strong performance in rendering and surface reconstruction.

Problem

• Improves rendering quality and 3D reconstruction accuracy simultaneously.
• Addresses challenges in surface reconstruction under diverse light conditions.
• Proposes a mutual-boosted approach using 2D/3D Gaussian splatting.

Innovation

• 2D/3D mutual-boosted Gaussian splatting for surface reconstruction
• Geometry-guided illumination decomposition for realistic rendering
• Alternating optimization with mutual supervision between branches