Video Generation Models Are Inherent Lighting Estimators

2026-07-06

AI Summary

Recovering dynamic high-dynamic-range (HDR) environmental illumination from a single in-the-wild video for photorealistic rendering remains challenging. This work reframes illumination estimation as a guided video inpainting task: by inserting synthetic chrome spheres into the scene, it leverages the spatiotemporal context modeling capabilities of video diffusion models to generate physically plausible reflections, thereby implicitly extracting lighting information. We demonstrate for the first time that modern video diffusion models possess an intrinsic physical understanding of illumination and introduce the V-LITE framework to harness this capability for practical use. To bridge the gap between low-dynamic-range (LDR) pretraining and HDR lighting estimation, we design an HDR-aware VAE, curate a hybrid dataset, and employ a LoRA-based fine-tuning strategy. Experiments show that our approach produces temporally coherent and physically plausible HDR environment maps, establishing video generative models not only as content synthesizers but also as powerful illumination estimators.

Abstract

Recovering dynamic environment maps from a single in-the-wild video is crucial for photorealistic rendering, yet remains a challenge. Recent video generation models can produce photorealistic scenes with complex lighting, possessing an inherent understanding of lighting. In this paper, we introduce V-LITE (Video generation models are inherent lighting estimators), a framework that unlocks this internal knowledge by reframing lighting estimation as a guided video inpainting task. Inspired by VFX industry practices, we insert a synthetic chrome ball into the scene to compel the model to generate physically plausible reflections from the surrounding spatio-temporal context. To bridge the gap from LDR-native models to the HDR domain, we design an HDR-aware VAE and employ an efficient LoRA-based fine-tuning strategy. We then construct a mixed dataset comprising high-fidelity HDR images to provide realistic HDR priors, and in-the-wild HDR videos to provide dynamic spatio-temporal context. Extensive experiments demonstrate that V-LITE produces temporally coherent HDR environment maps, revealing that modern video diffusion models are not merely synthesizers but also powerful, inherently capable estimators of physical scene lighting.

Problem

• lighting estimation
• HDR environment maps
• video generation
• dynamic lighting
• photorealistic rendering

Innovation

• lighting estimation
• video inpainting
• HDR reconstruction
• diffusion models
• LoRA fine-tuning