PIXLRelight: Controllable Relighting via Intrinsic Conditioning

2026-05-18

AI Summary

Existing single-image relighting methods suffer from limited lighting control, error accumulation in cascaded pipelines, or the need for per-image optimization. This work proposes a feed-forward relighting framework that, for the first time, integrates intrinsic image decomposition with neural rendering. By sharing intrinsic cues—such as albedo, diffuse shading, and non-diffuse residuals—it bridges physics-based rendering and learned synthesis, enabling arbitrary physically based rendering (PBR)-compatible lighting edits. The approach leverages a Transformer-based neural renderer, path-tracing-guided coarse 3D reconstruction, and pixel-wise affine modulation to achieve high-quality relighting at unprecedented speed (<0.1 seconds per image), while preserving physical plausibility and fine details. The method attains state-of-the-art performance across standard benchmarks.

Abstract

We present PIXLRelight, a feed-forward approach for physically controllable single-image relighting. Existing methods either provide limited lighting control (e.g. through text or environment maps), accumulate errors when chaining inverse and forward rendering, or require costly per-image optimization. Our key idea is to bridge physically based rendering (PBR) and learned image synthesis through a shared intrinsic conditioning that can be obtained from either real photographs or PBR renders. At training time, paired multi-illumination photographs are decomposed into albedo, diffuse shading, and non-diffuse residuals, which condition the model. At inference time, the same conditioning is computed from a path-traced render of a coarse 3D reconstruction of the input under user-specified PBR lights. A transformer-based neural renderer then applies the target illumination to the source photograph, preserving fine image detail through a per-pixel affine modulation. PIXLRelight enables arbitrary PBR-style lighting control, achieves state-of-the-art relighting quality, and runs in under a tenth of a second per image. Code and models are available at https://mlfarinha.github.io/pixl-relight/.

Problem

• relighting
• lighting control
• physically based rendering
• single-image relighting
• image synthesis

Innovation

• intrinsic conditioning
• physically based rendering
• neural relighting
• transformer-based renderer
• single-image relighting