Differentiable Light Transport with Gaussian Surfels via Adapted Radiosity for Efficient Relighting and Geometry Reconstruction

2025-09-22

AI Summary

Existing radiance field methods struggle to jointly model material reflectance and relighting, resulting in geometric blurring and low relighting efficiency. This paper proposes a differentiable light transport framework based on Gaussian surface elements and an enhanced radiosity theory: it jointly represents diffuse and specular reflection in spherical harmonic (SH) coefficient space; incorporates non-binary visibility and translucent primitives to extend the classical radiosity model; and introduces an efficient solver with analytically derived gradients to enable end-to-end optimization. The framework adopts Gaussian surface elements as geometric primitives and spherical harmonics as illumination encodings, enabling unified, differentiable inverse rendering for geometry reconstruction, novel-view synthesis, and relighting. Experiments demonstrate significant improvements over state-of-the-art methods under sparse input conditions and support real-time global illumination rendering at hundreds of frames per second.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Mixed Discrete/Continuous Search
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Radiance fields have gained tremendous success with applications ranging from novel view synthesis to geometry reconstruction, especially with the advent of Gaussian splatting. However, they sacrifice modeling of material reflective properties and lighting conditions, leading to significant geometric ambiguities and the inability to easily perform relighting. One way to address these limitations is to incorporate physically-based rendering, but it has been prohibitively expensive to include full global illumination within the inner loop of the optimization. Therefore, previous works adopt simplifications that make the whole optimization with global illumination effects efficient but less accurate. In this work, we adopt Gaussian surfels as the primitives and build an efficient framework for differentiable light transport, inspired from the classic radiosity theory. The whole framework operates in the coefficient space of spherical harmonics, enabling both diffuse and specular materials. We extend the classic radiosity into non-binary visibility and semi-opaque primitives, propose novel solvers to efficiently solve the light transport, and derive the backward pass for gradient optimizations, which is more efficient than auto-differentiation. During inference, we achieve view-independent rendering where light transport need not be recomputed under viewpoint changes, enabling hundreds of FPS for global illumination effects, including view-dependent reflections using a spherical harmonics representation. Through extensive qualitative and quantitative experiments, we demonstrate superior geometry reconstruction, view synthesis and relighting than previous inverse rendering baselines, or data-driven baselines given relatively sparse datasets with known or unknown lighting conditions.

Problem

• Addresses geometric ambiguities in radiance fields from material and lighting limitations
• Enables efficient differentiable global illumination for inverse rendering optimization
• Achieves real-time relighting without recomputing light transport for viewpoint changes

Innovation

• Gaussian surfels with adapted radiosity framework
• Spherical harmonics coefficient space for materials
• Efficient differentiable light transport solvers