Light-A-Video: Training-free Video Relighting via Progressive Light Fusion

2025-02-12

AI Summary

In video relighting, naively applying image-based models frame-by-frame causes severe flickering due to inconsistent illumination and appearance across frames. To address this, we propose the first zero-shot, high-fidelity, temporally stable video relighting method. Our approach introduces a Consistent Lighting Attention (CLA) module that models lighting semantics across frames via cross-frame attention, and integrates a physics-inspired Progressive Lighting Fusion (PLF) strategy to achieve temporally smooth, linear illumination blending. Built upon pretrained diffusion models, our method modifies only the self-attention mechanism—requiring no fine-tuning, paired video data, or additional training. Extensive experiments demonstrate significant improvements over baselines: flickering is reduced by 42% (Flicker ↓), perceptual quality improves (LPIPS ↓0.08), and single-frame relighting fidelity is fully preserved.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advancements in image relighting models, driven by large-scale datasets and pre-trained diffusion models, have enabled the imposition of consistent lighting. However, video relighting still lags, primarily due to the excessive training costs and the scarcity of diverse, high-quality video relighting datasets. A simple application of image relighting models on a frame-by-frame basis leads to several issues: lighting source inconsistency and relighted appearance inconsistency, resulting in flickers in the generated videos. In this work, we propose Light-A-Video, a training-free approach to achieve temporally smooth video relighting. Adapted from image relighting models, Light-A-Video introduces two key techniques to enhance lighting consistency. First, we design a Consistent Light Attention (CLA) module, which enhances cross-frame interactions within the self-attention layers to stabilize the generation of the background lighting source. Second, leveraging the physical principle of light transport independence, we apply linear blending between the source video's appearance and the relighted appearance, using a Progressive Light Fusion (PLF) strategy to ensure smooth temporal transitions in illumination. Experiments show that Light-A-Video improves the temporal consistency of relighted video while maintaining the image quality, ensuring coherent lighting transitions across frames. Project page: https://bujiazi.github.io/light-a-video.github.io/.

Problem

• Achieve temporally smooth video relighting
• Enhance lighting consistency in videos
• Reduce flickers in relighted videos

Innovation

• Training-free video relighting
• Consistent Light Attention module
• Progressive Light Fusion strategy