LumiCtrl : Learning Illuminant Prompts for Lighting Control in Personalized Text-to-Image Models

2025-12-19

AI Summary

Existing text-to-image (T2I) models struggle to precisely control scene illumination, limiting controllable generation of emotion, atmosphere, and aesthetics. To address this, we propose the first physics-guided framework for single-image illumination customization. Our method comprises three key components: (i) physically grounded illumination enhancement based on Planckian locus modeling to improve illumination fidelity; (ii) edge-guided illumination-structure disentangled prompt learning, which freezes ControlNet parameters to enable structure-aware illumination modeling; and (iii) foreground-focused + background-adaptive masked reconstruction loss for context-aware illumination transfer. Requiring only a single object image, our approach learns dedicated illumination prompts end-to-end. Extensive evaluations demonstrate significant improvements over state-of-the-art customization methods in illumination accuracy, visual appeal, and scene consistency. User preference studies further validate its effectiveness and perceptual superiority.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Current text-to-image (T2I) models have demonstrated remarkable progress in creative image generation, yet they still lack precise control over scene illuminants, which is a crucial factor for content designers aiming to manipulate the mood, atmosphere, and visual aesthetics of generated images. In this paper, we present an illuminant personalization method named LumiCtrl that learns an illuminant prompt given a single image of an object. LumiCtrl consists of three basic components: given an image of the object, our method applies (a) physics-based illuminant augmentation along the Planckian locus to create fine-tuning variants under standard illuminants; (b) edge-guided prompt disentanglement using a frozen ControlNet to ensure prompts focus on illumination rather than structure; and (c) a masked reconstruction loss that focuses learning on the foreground object while allowing the background to adapt contextually, enabling what we call contextual light adaptation. We qualitatively and quantitatively compare LumiCtrl against other T2I customization methods. The results show that our method achieves significantly better illuminant fidelity, aesthetic quality, and scene coherence compared to existing personalization baselines. A human preference study further confirms strong user preference for LumiCtrl outputs. The code and data will be released upon publication.

Problem

• Enables precise lighting control in text-to-image generation
• Learns illuminant prompts from a single object image
• Improves illuminant fidelity, aesthetics, and scene coherence

Innovation

• Physics-based illuminant augmentation for fine-tuning
• Edge-guided prompt disentanglement using frozen ControlNet
• Masked reconstruction loss for contextual light adaptation