Learning Illumination Control in Diffusion Models

2026-04-27

AI Summary

This work addresses the limitations of existing open-source methods for image illumination control, which often rely on complex inputs such as depth maps or lack publicly available data and code, hindering reproducibility. The authors propose the first end-to-end, open-source diffusion-based approach that requires only natural language instructions—without any geometric or lighting priors—to manipulate image illumination. A custom data engine automatically constructs training triplets comprising underexposed images, natural language prompts, and target well-lit images, enabling fine-tuning of Stable Diffusion variants (SD 1.5, SDXL, and FLUX.1-dev). Experimental results demonstrate that the method significantly outperforms current baselines in perceptual similarity, structural consistency, and identity preservation, achieving high-quality, controllable illumination editing. The dataset, code, and model weights are fully released to support reproducibility and future research.

Abstract

Controlling illumination in images is essential for photography and visual content creation. While closed-source models have demonstrated impressive illumination control, open-source alternatives either require heavy control inputs like depth maps or do not release their data and code. We present a fully open-source and reproducible pipeline for learning illumination control in diffusion models. Our approach builds a data engine that transforms well-lit images into supervised training triplets consisting of a poorly-illuminated input image, a natural language lighting instruction, and a well-illuminated output image. We finetune a diffusion model on this data and demonstrate significant improvements over baseline SD 1.5, SDXL, and FLUX.1-dev models in perceptual similarity, structural similarity, and identity preservation. Our work provides a reproducible solution built entirely with open-source tools and publicly available data. We release all our code, data, and model weights publicly.

Problem

• illumination control
• diffusion models
• open-source
• image editing
• reproducibility

Innovation

• illumination control
• diffusion models
• open-source pipeline
• supervised training triplets
• natural language conditioning