BRDFusion: Physics Meets Generation for Urban Scene Inverse Rendering

2026-06-15

AI Summary

This work addresses the limitations of existing approaches in urban street-view inverse rendering, where physics-based methods often produce reconstruction and rendering artifacts, while purely generative models lack geometric and lighting consistency as well as editability. To overcome these challenges, we propose BRDFusion—the first unified framework that deeply integrates physically driven inverse rendering with generative priors. Our method leverages physical modeling to recover explicit and consistent scene attributes such as BRDF and illumination, while incorporating generative priors to resolve optimization ambiguities and denoise artifacts during forward rendering. By synergistically combining physically based rendering (PBR), inverse rendering optimization, and neural image synthesis, BRDFusion outperforms state-of-the-art methods on both real-world and synthetic urban scenes, enabling high-quality novel view synthesis, relighting, nighttime simulation, and dynamic object insertion and editing.

Abstract

Inverse rendering of urban scenes from captured videos enables numerous applications, including content creation and autonomous driving simulation. Physically-based rendering methods follow and control lighting physics, but suffer from reconstruction and rendering artifacts. While generative models produce realistic videos, they offer limited consistency and controllability. We present BRDFusion, a unified framework that combines two complementary models for inverse and forward rendering. Specifically, BRDFusion recovers explicit, consistent scene properties with physical modeling and alleviates optimization ambiguity with generative priors. During forward rendering, the physical model provides controllable rendering from the scene configuration, and the generative model denoises and fixes artifacts. Therefore, our method produces high-quality videos while allowing precise control, outperforming baselines in real and synthetic scenes. Moreover, BRDFusion supports novel-view relighting, night simulation, and dynamic object insertion/editing. Project page: https://shigon255.github.io/brdfusion-page/

Problem

• inverse rendering
• urban scenes
• physically-based rendering
• generative models
• scene reconstruction

Innovation

• inverse rendering
• physically-based rendering
• generative priors
• urban scene
• BRDFusion