PIR: Photometric Inverse Rendering with Shading Cues Modeling and Surface Reflectance Regularization

2024-08-13

AI Summary

This work addresses three key limitations in single-image photometric inverse rendering: inadequate self-shadow modeling, neglect of interreflections, and inaccurate decoupling of albedo and illumination. To this end, we propose a joint optimization framework comprising: (1) explicit light source localization for physically accurate self-shadow generation; (2) DINO-based visual feature distillation as a material consistency regularizer to mitigate the ill-posedness of albedo–illumination decomposition; and (3) differentiable rendering integrated with importance sampling and end-to-end neural optimization. Extensive experiments on both synthetic and real-world datasets demonstrate that our method significantly outperforms existing state-of-the-art approaches. Notably, it achieves substantial improvements in albedo reconstruction accuracy—particularly in shadowed regions and specular highlights—while also recovering more faithful geometry and illumination.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Search and Optimization: Local Search
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Algorithmic accountability and transparency on the web
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

This paper addresses the problem of inverse rendering from photometric images. Existing approaches for this problem suffer from the effects of self-shadows, inter-reflections, and lack of constraints on the surface reflectance, leading to inaccurate decomposition of reflectance and illumination due to the ill-posed nature of inverse rendering. In this work, we propose a new method for neural inverse rendering. Our method jointly optimizes the light source position to account for the self-shadows in images, and computes indirect illumination using a differentiable rendering layer and an importance sampling strategy. To enhance surface reflectance decomposition, we introduce a new regularization by distilling DINO features to foster accurate and consistent material decomposition. Extensive experiments on synthetic and real datasets demonstrate that our method outperforms the state-of-the-art methods in reflectance decomposition.

Problem

• Photorealistic Rendering
• Material Inference
• Lighting Estimation

Innovation

• Neural Network
• DINO Technology
• Material Recognition