GAINS: Gaussian-based Inverse Rendering from Sparse Multi-View Captures

2025-12-10

AI Summary

In sparse multi-view Gaussian Splatting-based inverse rendering, severe coupling among geometry, material, and illumination leads to significant material reconstruction artifacts. To address this, we propose a two-stage decoupling framework: (1) geometry optimization via fusion of monocular depth/normal estimation and diffusion priors; (2) material estimation regularized by semantic segmentation, intrinsic image decomposition, diffusion priors, and differentiable light transport modeling. This work is the first to embed diffusion model priors into Gaussian Splatting inverse rendering, synergistically integrating semantic priors and physical constraints for joint geometry reconstruction and material decoupling. Evaluated on both synthetic and real-world datasets, our method reduces material parameter error by 32%, improves relighting PSNR by 4.1 dB, and increases novel-view synthesis SSIM by 0.08—substantially outperforming state-of-the-art approaches.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Mixed Discrete/Continuous Search
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent advances in Gaussian Splatting-based inverse rendering extend Gaussian primitives with shading parameters and physically grounded light transport, enabling high-quality material recovery from dense multi-view captures. However, these methods degrade sharply under sparse-view settings, where limited observations lead to severe ambiguity between geometry, reflectance, and lighting. We introduce GAINS (Gaussian-based Inverse rendering from Sparse multi-view captures), a two-stage inverse rendering framework that leverages learning-based priors to stabilize geometry and material estimation. GAINS first refines geometry using monocular depth/normal and diffusion priors, then employs segmentation, intrinsic image decomposition (IID), and diffusion priors to regularize material recovery. Extensive experiments on synthetic and real-world datasets show that GAINS significantly improves material parameter accuracy, relighting quality, and novel-view synthesis compared to state-of-the-art Gaussian-based inverse rendering methods, especially under sparse-view settings. Project page: https://patrickbail.github.io/gains/

Problem

• Addresses material recovery ambiguity in sparse-view captures
• Introduces a two-stage framework with learning-based priors
• Improves accuracy and quality in inverse rendering under limited observations

Innovation

• Two-stage framework with learning-based priors
• Geometry refinement using monocular depth and diffusion
• Material regularization via segmentation and intrinsic decomposition