Learning Latent Proxies for Controllable Single-Image Relighting

2026-03-16

AI Summary

Single-image relighting is highly ill-posed due to the absence of explicit geometry and material information, making fine-grained and controllable editing challenging. This work proposes LightCtrl, a method that circumvents full intrinsic decomposition by leveraging minimal physics-based rendering (PBR) supervision to extract compact latent proxies encoding material and geometric cues. These proxies, combined with an illumination-sensitive region mask, guide a diffusion model for realistic relighting. By incorporating sparse yet physically plausible latent cues, a newly curated large-scale dataset named ScaLight, and a direct preference optimization (DPO) strategy, LightCtrl significantly enhances both physical consistency and control accuracy. Experiments demonstrate consistent superiority over state-of-the-art methods on both object- and scene-level benchmarks, achieving up to a 2.4 dB gain in PSNR and a 35% reduction in RMSE.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Single-image relighting is highly under-constrained: small illumination changes can produce large, nonlinear variations in shading, shadows, and specularities, while geometry and materials remain unobserved. Existing diffusion-based approaches either rely on intrinsic or G-buffer pipelines that require dense and fragile supervision, or operate purely in latent space without physical grounding, making fine-grained control of direction, intensity, and color unreliable. We observe that a full intrinsic decomposition is unnecessary and redundant for accurate relighting. Instead, sparse but physically meaningful cues, indicating where illumination should change and how materials should respond, are sufficient to guide a diffusion model. Based on this insight, we introduce LightCtrl that integrates physical priors at two levels: a few-shot latent proxy encoder that extracts compact material-geometry cues from limited PBR supervision, and a lighting-aware mask that identifies sensitive illumination regions and steers the denoiser toward shading relevant pixels. To compensate for scarce PBR data, we refine the proxy branch using a DPO-based objective that enforces physical consistency in the predicted cues. We also present ScaLight, a large-scale object-level dataset with systematically varied illumination and complete camera-light metadata, enabling physically consistent and controllable training. Across object and scene level benchmarks, our method achieves photometrically faithful relighting with accurate continuous control, surpassing prior diffusion and intrinsic-based baselines, including gains of up to +2.4 dB PSNR and 35% lower RMSE under controlled lighting shifts.

Problem

• single-image relighting
• illumination control
• physical consistency
• under-constrained problem
• material-geometry cues

Innovation

• latent proxies
• controllable relighting
• diffusion models
• physical priors
• single-image relighting