RTR-GS: 3D Gaussian Splatting for Inverse Rendering with Radiance Transfer and Reflection

2025-07-10

AI Summary

To address the coupling between BRDF and illumination, as well as high-frequency detail distortion in inverse rendering and relighting of reflective objects, this paper proposes the first 3D Gaussian Splatting (3DGS)-based inverse rendering framework supporting arbitrary reflectance materials. Methodologically, it introduces a hybrid dual-branch architecture: a forward radiative transfer branch using spherical harmonics to model global illumination and reconstruct geometry, and a physically inspired deferred reflection rendering branch that explicitly decouples BRDF from incident lighting, enabling end-to-end optimization via differentiable rendering. Crucially, it pioneers the integration of 3D Gaussian rasterization for reflective object modeling and mitigates floating artifacts—caused by spherical harmonics’ high-frequency overfitting—through cross-branch regularization. Experiments demonstrate state-of-the-art performance in novel-view synthesis, normal/material/illumination decomposition, and relighting, while maintaining efficient training and inference.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Mixed Discrete/Continuous Search
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Sustainability and carbon-aware systems for Web, mobile, and WoT
• Security and Privacy: Data transparency and provenance

Abstract

3D Gaussian Splatting (3DGS) has demonstrated impressive capabilities in novel view synthesis. However, rendering reflective objects remains a significant challenge, particularly in inverse rendering and relighting. We introduce RTR-GS, a novel inverse rendering framework capable of robustly rendering objects with arbitrary reflectance properties, decomposing BRDF and lighting, and delivering credible relighting results. Given a collection of multi-view images, our method effectively recovers geometric structure through a hybrid rendering model that combines forward rendering for radiance transfer with deferred rendering for reflections. This approach successfully separates high-frequency and low-frequency appearances, mitigating floating artifacts caused by spherical harmonic overfitting when handling high-frequency details. We further refine BRDF and lighting decomposition using an additional physically-based deferred rendering branch. Experimental results show that our method enhances novel view synthesis, normal estimation, decomposition, and relighting while maintaining efficient training inference process.

Problem

• Rendering reflective objects in inverse rendering
• Decomposing BRDF and lighting accurately
• Mitigating artifacts from spherical harmonic overfitting

Innovation

• Hybrid rendering model combining forward and deferred rendering
• Separates high-frequency and low-frequency appearances effectively
• Refines BRDF and lighting decomposition with deferred rendering