SIR: Multi-view Inverse Rendering with Decomposable Shadow Under Indoor Intense Lighting

2024-02-09

AI Summary

In indoor scenes with strong illumination, joint decomposition of material and lighting remains challenging; shadow fidelity is low; and unknown light source positions cause distortions in material reconstruction. Method: This paper proposes a high-fidelity inverse rendering approach leveraging multi-view HDR images. It employs an SDF-based neural radiance field as geometric prior and introduces a differentiable shadow decomposition mechanism—first of its kind—integrated with a three-stage SVBRDF estimation pipeline regularized by BRDF-physical constraints to achieve robust lighting-material decoupling. Contribution/Results: The method significantly improves material consistency in complex shadowed regions via differentiable shadow modeling. Evaluated on both synthetic and real indoor datasets, it outperforms state-of-the-art methods quantitatively and qualitatively, enabling high-quality editing tasks including novel-view relighting, object insertion, and material replacement.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: State Estimation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Systems and Infrastructure for Web, Mobile and WoT: Sustainability and carbon-aware systems for Web, mobile, and WoT

Abstract

We propose SIR, an efficient method to decompose differentiable shadows for inverse rendering on indoor scenes using multi-view data, addressing the challenges in accurately decomposing the materials and lighting conditions. Unlike previous methods that struggle with shadow fidelity in complex lighting environments, our approach explicitly learns shadows for enhanced realism in material estimation under unknown light positions. Utilizing posed HDR images as input, SIR employs an SDF-based neural radiance field for comprehensive scene representation. Then, SIR integrates a shadow term with a three-stage material estimation approach to improve SVBRDF quality. Specifically, SIR is designed to learn a differentiable shadow, complemented by BRDF regularization, to optimize inverse rendering accuracy. Extensive experiments on both synthetic and real-world indoor scenes demonstrate the superior performance of SIR over existing methods in both quantitative metrics and qualitative analysis. The significant decomposing ability of SIR enables sophisticated editing capabilities like free-view relighting, object insertion, and material replacement. The code and data are available at https://xiaokangwei.github.io/SIR/.

Problem

• Decompose shadows for accurate inverse rendering indoors
• Improve material estimation under unknown lighting conditions
• Enhance SVBRDF quality using multi-view HDR inputs

Innovation

• Decomposes differentiable shadows for inverse rendering
• Uses SDF-based neural radiance field representation
• Integrates shadow term with three-stage material estimation