Relit-LiVE: Relight Video by Jointly Learning Environment Video

2026-05-07

AI Summary

Existing video relighting methods rely on unstable intrinsic decomposition, often causing appearance distortions, material degradation, and temporal artifacts in real-world videos. This work proposes a novel video relighting framework that operates without prior camera pose information, uniquely integrating the original reference image into the diffusion-based rendering process. The method jointly predicts both the relit video and a viewpoint-aligned environmental illumination video within a single forward pass of a large-scale video diffusion model, ensuring geometric consistency, physical plausibility, and temporal stability. Experimental results demonstrate that the approach outperforms state-of-the-art methods on both synthetic and real datasets, while enabling diverse downstream applications such as scene rendering, material editing, object insertion, and streaming processing.

Abstract

Recent advances have shown that large-scale video diffusion models can be repurposed as neural renderers by first decomposing videos into intrinsic scene representations and then performing forward rendering under novel illumination. While promising, this paradigm fundamentally relies on accurate intrinsic decomposition, which remains highly unreliable for real-world videos and often leads to distorted appearances, broken materials, and accumulated temporal artifacts during relighting. In this work, we present Relit-LiVE, a novel video relighting framework that produces physically consistent, temporally stable results without requiring prior knowledge of camera pose. Our key insight is to explicitly introduce raw reference images into the rendering process, enabling the model to recover critical scene cues that are inevitably lost or corrupted in intrinsic representations. Furthermore, we propose a novel environment video prediction formulation that simultaneously generates relit videos and per-frame environment maps aligned with each camera viewpoint in a single diffusion process. This joint prediction enforces strong geometric-illumination alignment and naturally supports dynamic lighting and camera motion, significantly improving physical consistency in video relighting while easing the requirement of known per-frame camera pose. Extensive experiments demonstrate that Relit-LiVE consistently outperforms state-of-the-art video relighting and neural rendering methods across synthetic and real-world benchmarks. Beyond relighting, our framework naturally supports a wide range of downstream applications, including scene-level rendering, material editing, object insertion, and streaming video relighting. The Project is available at https://github.com/zhuxing0/Relit-LiVE.

Problem

• video relighting
• intrinsic decomposition
• temporal artifacts
• physical consistency
• neural rendering

Innovation

• video relighting
• diffusion models
• intrinsic decomposition
• environment map prediction
• neural rendering