Diffusion-Based Material Regularization for Physics-Based Inverse Rendering

2026-06-29

AI Summary

Existing inverse rendering methods suffer from ill-posedness, leading to entangled material and lighting estimates and poor generalization. While diffusion models can generate visually plausible materials, their outputs often violate the physical rendering equation. This work proposes a novel approach that integrates physics-based inverse rendering with diffusion priors by transforming the output of a pretrained diffusion model into a spatially adaptive regularizer. This regularizer guides optimization in smooth material regions while preserving accurate fitting to multi-view images. Implemented within a differentiable rendering framework, the method enables end-to-end joint reconstruction of geometry, materials, and lighting, effectively harmonizing physical accuracy with visual plausibility for the first time. Experiments on Synthetic4Relight, Stanford-ORB, and DTC-Synthetic datasets demonstrate significant improvements over state-of-the-art methods in both reconstruction accuracy and relighting quality, with generated assets directly compatible with standard rendering pipelines.

Abstract

Reconstructing physics-based 3D assets -- geometry, materials, and illumination -- from multi-view images is a core problem in computer graphics and vision, and a prerequisite for realistic relighting and editing. Physics-based inverse rendering offers an accurate image-formation model, but is severely underconstrained: without strong priors, illumination is baked into materials, and reconstructions generalize poorly to novel views and lighting. Data-driven diffusion models, in contrast, predict visually plausible materials, yet their predictions rarely satisfy the rendering equation and are not directly usable for physics-based rendering. We bridge these two paradigms rather than replacing either. Our key idea is to treat the predictions of a state-of-the-art diffusion model not as target material values but as a similarity kernel for optimization: we introduce a regularization loss that penalizes deviations in the optimized material over surface regions where the diffusion predictions are near-constant, while leaving the optimization free to match the input images. Built on this regularizer, our end-to-end pipeline jointly reconstructs geometry, materials, and illumination, yielding high-quality assets that drop into standard rendering pipelines and relight faithfully. On the Synthetic4Relight, Stanford-ORB, and DTC-Synthetic datasets, our method significantly outperforms state-of-the-art baselines in both reconstruction accuracy and relighting quality.

Problem

• inverse rendering
• material reconstruction
• diffusion models
• physics-based rendering
• relighting

Innovation

• diffusion-based regularization
• physics-based inverse rendering
• material reconstruction
• relighting
• differentiable rendering