HarmoVid: Relightful Video Portrait Harmonization

2026-05-27

AI Summary

Existing methods for video portrait relighting often suffer from temporal flickering and incoherence due to the lack of paired real-world data. To address this, this work proposes a deflickering illumination model that generates temporally stable paired videos for training. By integrating a hybrid strategy combining real and synthetic videos, we develop a video diffusion model enhanced with an asymmetric Alpha mask conditioning mechanism to improve boundary sharpness and temporal consistency. The proposed approach maintains strong relighting capabilities while significantly outperforming current image- and video-level methods, achieving notable improvements in temporal stability, visual naturalness, boundary quality, and physically plausible lighting effects.

Abstract

We present a method for harmonizing the lighting of a foreground video to match a target background scene, adjusting shadows, color tone, and illumination intensity (relightful harmonization). Unlike images, acquiring labeled data for videos, where identical motions are recorded under different lighting conditions, is practically infeasible and non-scalable. While one way to create such paired data is to apply existing image-based harmonization models frame by frame to a video, the resulting outputs often suffer from significant temporal jitters. We overcome this problem by introducing a novel lighting deflickering model that can stabilize the global and local lighting flickering artifacts. Our video diffusion model learns from these upgraded deflickered data with a volume of real and synthetic videos to generate high-quality video harmonization results. We further propose an asymmetric alpha mask conditioning technique to learn the clean boundaries from real videos. Experiments demonstrate that our model achieves strong temporal coherence, naturalness, cleaner boundaries, and physically meaningful lighting behavior, while maintaining strong relighting expressiveness compared to prior image-based and video-based harmonization methods.

Problem

• video harmonization
• relighting
• temporal coherence
• lighting flickering
• portrait harmonization

Innovation

• video harmonization
• lighting deflickering
• diffusion model
• temporal coherence
• asymmetric alpha mask