RNG: Relightable Neural Gaussians

2024-09-29

arXiv.org

AI Summary

Existing 3D Gaussian Splatting (3DGS) struggles with free-viewpoint relighting of soft-boundary objects—such as hair and fabrics—due to severe geometry-appearance-illumination coupling and reliance on strong shadow priors. To address this, we propose a shading-model-free disentangled representation: (i) a novel radiance field jointly conditioned on view and light directions; (ii) a differentiable shadow-guided depth refinement network; and (iii) a forward-delayed hybrid optimization strategy. Integrated within the 3DGS framework, our method combines directional-conditioned radiance modeling, shadow-aware loss, and depth refinement, enabling real-time rendering at 60 FPS after only 1.3 hours of training. Experiments demonstrate significantly improved shadow fidelity over state-of-the-art 3DGS methods, while outperforming NeRF-based baselines in both speed and reconstruction quality.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Learning with Manifolds

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

3D Gaussian Splatting (3DGS) has shown impressive results for the novel view synthesis task, where lighting is assumed to be fixed. However, creating relightable 3D assets, especially for objects with ill-defined shapes (fur, fabric, etc.), remains a challenging task. The decomposition between light, geometry, and material is ambiguous, especially if either smooth surface assumptions or surfacebased analytical shading models do not apply. We propose Relightable Neural Gaussians (RNG), a novel 3DGS-based framework that enables the relighting of objects with both hard surfaces or soft boundaries, while avoiding assumptions on the shading model. We condition the radiance at each point on both view and light directions. We also introduce a shadow cue, as well as a depth refinement network to improve shadow accuracy. Finally, we propose a hybrid forward-deferred fitting strategy to balance geometry and appearance quality. Our method achieves significantly faster training (1.3 hours) and rendering (60 frames per second) compared to a prior method based on neural radiance fields and produces higher-quality shadows than a concurrent 3DGS-based method. Project page: https://www.whois-jiahui.fun/project_pages/RNG.

Problem

• Enables relighting of objects with hard or soft surfaces
• Avoids shading model assumptions for ambiguous geometry
• Balances geometry and appearance quality efficiently

Innovation

• Relightable 3DGS framework for hard and soft surfaces
• View and light direction conditioned radiance
• Hybrid forward-deferred fitting strategy