Inverse Rendering using Multi-Bounce Path Tracing and Reservoir Sampling

2024-06-24

AI Summary

This work addresses joint inverse rendering of geometry, material, and illumination from multi-view images, with particular emphasis on modeling complex shadows (e.g., self-shadowing) and indirect illumination (e.g., internal reflections). We propose a two-stage explicit framework: first reconstructing a triangle-mesh geometry, then building a physically consistent, differentiable inverse renderer via multi-bounce path tracing coupled with reservoir sampling. To our knowledge, this is the first use of reservoir sampling in multi-bounce Monte Carlo integration for stable gradient-based optimization under low sample counts. Our method explicitly models indirect illumination and enables end-to-end joint optimization. It achieves state-of-the-art decomposition accuracy on challenging shadowed scenes, produces editable explicit meshes, and supports real-time relighting, material editing, and seamless integration with CAD and graphics engines.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Search and Optimization: Sampling/Simulation-based Search
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We present MIRReS, a novel two-stage inverse rendering framework that jointly reconstructs and optimizes the explicit geometry, material, and lighting from multi-view images. Unlike previous methods that rely on implicit irradiance fields or simplified path tracing algorithms, our method extracts an explicit geometry (triangular mesh) in stage one, and introduces a more realistic physically-based inverse rendering model that utilizes multi-bounce path tracing and Monte Carlo integration. By leveraging multi-bounce path tracing, our method effectively estimates indirect illumination, including self-shadowing and internal reflections, which improves the intrinsic decomposition of shape, material, and lighting. Moreover, we incorporate reservoir sampling into our framework to address the noise in Monte Carlo integration, enhancing convergence and facilitating gradient-based optimization with low sample counts. Through qualitative and quantitative evaluation of several scenarios, especially in challenging scenarios with complex shadows, we demonstrate that our method achieves state-of-the-art performance on decomposition results. Additionally, our optimized explicit geometry enables applications such as scene editing, relighting, and material editing with modern graphics engines or CAD software. The source code is available at https://brabbitdousha.github.io/MIRReS/

Problem

• Reconstructs explicit geometry, material, lighting
• Estimates indirect illumination accurately
• Reduces noise in Monte Carlo integration

Innovation

• Multi-bounce path tracing
• Reservoir sampling
• Explicit geometry optimization