SRUG: Shadow-Guided Relightable Urban Scene with Generation Model

2026-05-23

AI Summary

Relighting urban scenes is highly challenging due to difficulties in modeling shadows in occluded regions, sparse input views, and ambiguities in material decomposition under complex illumination. This work proposes the SRUG framework, which innovatively leverages shadow cues to guide 3D scene completion for recovering unseen geometry and introduces a large material model (LMM)-driven iterative material decomposition strategy. Within a physically consistent lighting model, geometry and materials are jointly optimized in an end-to-end manner. The method significantly outperforms existing approaches on both novel view synthesis and relighting tasks, with quantitative metrics and visual evaluations consistently demonstrating its superior physical plausibility and robustness.

Abstract

Creating relightable urban scenes from images or videos is widely useful but highly ill-posed. Urban environments are typically unbounded and extend beyond the visible regions. As a result, many portions of the scene remain unobserved, yet these invisible regions can cast shadows onto visible areas. Reasonably modeling shadows cast by such invisible regions is challenging and poses a significant obstacle to creating relightable urban scenes. At the same time, sparse input views and complex illumination conditions further complicate relighting, as they introduce severe ambiguities in material decomposition. In this paper, we propose Shadow-guided Relightable Urban Scene with Generation model (SRUG), a novel framework designed to address relighting challenges in urban scenes. SRUG leverages shadows to guide a 3D completion model for recovering the geometry of invisible regions, promoting the synthesis of physically reasonable shadows. In addition, SRUG employs an iterative material decomposition scheme that applies the large material model (LMM) to provide material supervision and iteratively decompose the scene's material properties, enabling robust material decomposition. Building upon these components, we introduce a physically-based lighting model that captures the complex illumination of urban scenes and supports reliable relighting. Extensive quantitative evaluations and visual comparisons demonstrate that our method outperforms existing approaches in both novel view synthesis and relighting tasks.

Problem

• relightable urban scene
• shadow modeling
• material decomposition
• 3D scene completion
• complex illumination

Innovation

• shadow-guided completion
• relightable urban scene
• iterative material decomposition
• large material model (LMM)
• physically-based relighting