GS-ROR$^2$: Bidirectional-guided 3DGS and SDF for Reflective Object Relighting and Reconstruction

2024-05-22

AI Summary

To address weak geometric constraints in 3D Gaussian Splatting (3DGS) and the high computational cost of ray marching plus poor sharp-detail modeling in Signed Distance Function (SDF)-based methods for reconstructing reflective objects, this paper proposes GS-SDF, a bidirectional collaborative framework. Our method jointly optimizes explicit 3DGS and implicit SDF representations through mutual supervision: (1) SDF-guided Gaussian optimization enforces depth and normal consistency to enhance geometric fidelity; (2) Gaussian-predicted normals refine the SDF implicit field to improve mesh quality; and (3) SDF-aware pruning eliminates floating-point artifacts. The framework integrates implicit SDF modeling, bidirectional supervision, deferred shading, and alpha compositing rendering. Experiments show that GS-SDF achieves high-fidelity geometry reconstruction, real-time relightable rendering, and produces high-quality meshes with physically plausible normals—outperforming unimodal approaches—while incurring only a 17% increase in training time.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Mixed Discrete/Continuous Search
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

3D Gaussian Splatting (3DGS) has shown a powerful capability for novel view synthesis due to its detailed expressive ability and highly efficient rendering speed. Unfortunately, creating relightable 3D assets and reconstructing faithful geometry with 3DGS is still problematic, particularly for reflective objects, as its discontinuous representation raises difficulties in constraining geometries. Volumetric signed distance field (SDF) methods provide robust geometry reconstruction, while the expensive ray marching hinders its real-time application and slows the training. Besides, these methods struggle to capture sharp geometric details. To this end, we propose to guide 3DGS and SDF bidirectionally in a complementary manner, including an SDF-aided Gaussian splatting for efficient optimization of the relighting model and a GS-guided SDF enhancement for high-quality geometry reconstruction. At the core of our SDF-aided Gaussian splatting is the mutual supervision of the depth and normal between blended Gaussians and SDF, which avoids the expensive volume rendering of SDF. Thanks to this mutual supervision, the learned blended Gaussians are well-constrained with a minimal time cost. As the Gaussians are rendered in a deferred shading mode, the alpha-blended Gaussians are smooth, while individual Gaussians may still be outliers, yielding floater artifacts. Therefore, we introduce an SDF-aware pruning strategy to remove Gaussian outliers located distant from the surface defined by SDF, avoiding floater issue. This way, our GS framework provides reasonable normal and achieves realistic relighting, while the mesh from depth is still problematic. Therefore, we design a GS-guided SDF refinement, which utilizes the blended normal from Gaussians to finetune SDF. With this enhancement, our method can further provide high-quality meshes for reflective objects at the cost of 17% extra training time.

Problem

• 3DGS struggles with relightable assets and geometry for reflective objects
• Volumetric SDF methods are slow and lack sharp geometric details
• Bidirectional guidance combines 3DGS and SDF for efficient relighting and reconstruction

Innovation

• Bidirectional-guided 3DGS and SDF integration
• SDF-aided Gaussian splatting for relighting
• GS-guided SDF refinement for geometry