Many-Worlds Inverse Rendering

2024-08-13

arXiv.org

AI Summary

In inverse rendering, surface visibility discontinuities induce gradient distortion and hinder optimization. To address this, we propose a “multi-world” voxelized representation that models conflicting geometric interpretations as optically isolated superpositions—bypassing conventional reliance on exponentially complex stochastic media modeling or high-cost visibility sampling. Building upon this representation, we derive a novel light transport law and a Monte Carlo differentiable rendering algorithm, enabling, for the first time, end-to-end differentiable physically based inverse rendering. Our method strictly preserves physical consistency while significantly accelerating convergence, reducing per-iteration computational cost, and achieving robust, efficient joint reconstruction of geometry and material under strong occlusion.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Search and Optimization: Mixed Discrete/Continuous Search
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance
• Responsible Web: Algorithmic accountability and transparency on the web

Abstract

Discontinuous visibility changes remain a major bottleneck when optimizing surfaces within a physically-based inverse renderer. Many previous works have proposed sophisticated algorithms and data structures to sample visibility silhouettes more efficiently. Our work presents another solution: instead of differentiating a tentative surface locally, we differentiate a volumetric perturbation of a surface. We refer this as a many-worlds representation because it models a non-interacting superposition of conflicting explanations (worlds) of the input dataset. Each world is optically isolated from others, leading to a new transport law that distinguishes our method from prior work based on exponential random media. The resulting Monte Carlo algorithm is simpler and more efficient than prior methods. We demonstrate that our method promotes rapid convergence, both in terms of the total iteration count and the cost per iteration.

Problem

• Optimizing surfaces with discontinuous visibility changes in inverse rendering
• Differentiating volumetric perturbations instead of local surface changes
• Proposing a simpler, more efficient Monte Carlo algorithm for convergence

Innovation

• Differentiates volumetric perturbation of surface
• Uses many-worlds representation for conflicting explanations
• Simpler and more efficient Monte Carlo algorithm