DreamLight: Towards Harmonious and Consistent Image Relighting

2025-06-17

AI Summary

This work addresses the challenge of inconsistent illumination, color, and light–object interactions when replacing backgrounds in foreground images via text- or image-driven editing. To this end, we propose a general-purpose image relighting method. Our approach introduces the Position-Guided Light Adapter (PGLA) to model multi-directional lighting priors, designs a Spectral Foreground Refinement (SFF) module for adaptive high-/low-frequency feature fusion, and unifies text/image input paradigms by leveraging diffusion models’ semantic and illumination priors. Key technical components include directional-biased mask attention, ambient-light query embeddings, multi-scale spectral reweighting, and semantic-alignment distillation. Quantitative evaluation demonstrates state-of-the-art performance across multiple metrics. User studies confirm significant improvements in perceptual naturalness and scene coherence. Moreover, the method supports unlimited text-prompt-driven generation of photorealistic relit images.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We introduce a model named DreamLight for universal image relighting in this work, which can seamlessly composite subjects into a new background while maintaining aesthetic uniformity in terms of lighting and color tone. The background can be specified by natural images (image-based relighting) or generated from unlimited text prompts (text-based relighting). Existing studies primarily focus on image-based relighting, while with scant exploration into text-based scenarios. Some works employ intricate disentanglement pipeline designs relying on environment maps to provide relevant information, which grapples with the expensive data cost required for intrinsic decomposition and light source. Other methods take this task as an image translation problem and perform pixel-level transformation with autoencoder architecture. While these methods have achieved decent harmonization effects, they struggle to generate realistic and natural light interaction effects between the foreground and background. To alleviate these challenges, we reorganize the input data into a unified format and leverage the semantic prior provided by the pretrained diffusion model to facilitate the generation of natural results. Moreover, we propose a Position-Guided Light Adapter (PGLA) that condenses light information from different directions in the background into designed light query embeddings, and modulates the foreground with direction-biased masked attention. In addition, we present a post-processing module named Spectral Foreground Fixer (SFF) to adaptively reorganize different frequency components of subject and relighted background, which helps enhance the consistency of foreground appearance. Extensive comparisons and user study demonstrate that our DreamLight achieves remarkable relighting performance.

Problem

• Achieving harmonious image relighting with aesthetic lighting and color consistency
• Overcoming limitations of text-based relighting compared to image-based methods
• Generating realistic light interaction effects between foreground and background

Innovation

• Unified input data format with diffusion model
• Position-Guided Light Adapter for direction-biased modulation
• Spectral Foreground Fixer enhances appearance consistency