LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting

2026-07-08

AI Summary

Video relighting demands both temporal consistency and physically plausible illumination modeling, yet existing approaches suffer from limitations in intrinsic attribute estimation, global illumination simulation, or controllability. This work proposes a hybrid pipeline that reframes relighting as diffusion-based refinement of a physically based rendering (PBR) proxy video: first generating a PBR proxy under target illumination, then enhancing its details using a video diffusion model (CogVideoX). By implicitly encoding lighting conditions, the method circumvents the need to directly learn complex illumination representations and incorporates photometric priors to improve global illumination recovery. Experiments demonstrate state-of-the-art performance on real-world benchmarks, and the authors further introduce a new synthetic benchmark for systematic evaluation, along with publicly released datasets, metrics, and code.

Abstract

Video relighting requires balancing long-form temporal consistency with a physically grounded understanding of light transport, which depends on accurate estimation of intrinsic scene properties such as materials, geometry, and illumination. Existing methods follow two paradigms: (1) reconstruct a video's photometric properties via inverse rendering and relight them to a target illumination via forward rendering, using physically-based rendering (PBR) or a neural renderer; these suffer from noisy reconstructions and struggle with hard-to-model effects such as global illumination. (2) Frame the task as generative video-to-video translation conditioned on relighting targets (a target environment map or text); this limits relighting control and temporal stability, since diffusion models struggle to translate long-form videos, and is constrained by the availability of input/relit training pairs. We propose LightCrafter, a hybrid pipeline that reformulates video relighting as video translation of a proxy video: rather than translating the input video directly to the target, we translate a PBR rendering of the input under the target illumination to the final target. This bakes illumination targets into the PBR proxy, removing the need to teach the diffusion model illumination concepts like environment maps, and enables more intricate lighting control while naturally providing long-form temporal consistency. We show PBR renders alone already outperform some prior art but struggle with effects like global illumination; to capture these, we leverage photometric priors in video generation models by post-training CogVideoX on synthetic video pairs and real-world unpaired videos. We outperform prior state-of-the-art on existing real-world relighting benchmarks and contribute a synthetic benchmark for further analysis. We will release our dataset, benchmark, metrics, and code.

Problem

• video relighting
• temporal consistency
• light transport
• intrinsic scene properties
• physically-based rendering

Innovation

• video relighting
• PBR-conditioned diffusion
• temporal consistency
• proxy video translation
• photometric priors