RelightMaster: Precise Video Relighting with Multi-plane Light Images

2025-11-09

AI Summary

Existing text-to-video models struggle with fine-grained, physically plausible video relighting due to limited textual expressiveness for lighting, absence of lighting-aware pretraining, and scarcity of real-world multi-illumination video data. To address this, we introduce RelightVideo—the first multi-condition dynamic video dataset explicitly designed for relighting—and propose Multi-plane Light Image (MPLI), a novel visual prompt that explicitly encodes 3D light source position, intensity, and color, enabling multi-light support and zero-shot generalization to unseen illuminations. We further design a Light Image Adapter to inject MPLI features losslessly into pretrained video DiT models without catastrophic forgetting. Leveraging Unreal Engine-synthesized data, video VAE compression, and the DiT architecture, our method achieves high-fidelity, controllable relighting while preserving spatiotemporal content consistency. Experiments demonstrate significant improvements over baselines in shadow synthesis, cross-illumination transfer, and other relighting tasks.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Recent advances in diffusion models enable high-quality video generation and editing, but precise relighting with consistent video contents, which is critical for shaping scene atmosphere and viewer attention, remains unexplored. Mainstream text-to-video (T2V) models lack fine-grained lighting control due to text's inherent limitation in describing lighting details and insufficient pre-training on lighting-related prompts. Additionally, constructing high-quality relighting training data is challenging, as real-world controllable lighting data is scarce. To address these issues, we propose RelightMaster, a novel framework for accurate and controllable video relighting. First, we build RelightVideo, the first dataset with identical dynamic content under varying precise lighting conditions based on the Unreal Engine. Then, we introduce Multi-plane Light Image (MPLI), a novel visual prompt inspired by Multi-Plane Image (MPI). MPLI models lighting via K depth-aligned planes, representing 3D light source positions, intensities, and colors while supporting multi-source scenarios and generalizing to unseen light setups. Third, we design a Light Image Adapter that seamlessly injects MPLI into pre-trained Video Diffusion Transformers (DiT): it compresses MPLI via a pre-trained Video VAE and injects latent light features into DiT blocks, leveraging the base model's generative prior without catastrophic forgetting. Experiments show that RelightMaster generates physically plausible lighting and shadows and preserves original scene content. Demos are available at https://wkbian.github.io/Projects/RelightMaster/.

Problem

• Achieving precise video relighting with consistent content control
• Overcoming text limitations for fine-grained lighting manipulation
• Addressing scarcity of high-quality relighting training datasets

Innovation

• Constructed RelightVideo dataset with varied lighting conditions
• Introduced Multi-plane Light Image for 3D lighting modeling
• Designed Light Image Adapter for Video Diffusion Transformers