Physically Controllable Relighting of Photographs

2025-08-07

AI Summary

This work addresses the challenge of physically plausible relighting of single real-world photographs, where explicit illumination parameters are absent and precise lighting control is thus infeasible. We propose the first method enabling physically controllable relighting from a single unstructured image. Our approach reconstructs a color-coded 3D mesh via monocular depth estimation and intrinsic image decomposition, then integrates path tracing—ensuring physical accuracy—with feedforward neural rendering—guaranteeing visual fidelity—to support interactive, 3D-space lighting parameter manipulation. Crucially, we introduce a self-supervised learning framework that jointly optimizes geometry, material, and lighting end-to-end, without requiring paired data or ground-truth annotations. Trained exclusively on unlabeled real images, our method produces results that are both physically consistent and photorealistic. This represents the first successful extension of explicit, physics-based lighting control—previously confined to controlled 3D graphics settings—to the challenging single-image relighting task.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We present a self-supervised approach to in-the-wild image relighting that enables fully controllable, physically based illumination editing. We achieve this by combining the physical accuracy of traditional rendering with the photorealistic appearance made possible by neural rendering. Our pipeline works by inferring a colored mesh representation of a given scene using monocular estimates of geometry and intrinsic components. This representation allows users to define their desired illumination configuration in 3D. The scene under the new lighting can then be rendered using a path-tracing engine. We send this approximate rendering of the scene through a feed-forward neural renderer to predict the final photorealistic relighting result. We develop a differentiable rendering process to reconstruct in-the-wild scene illumination, enabling self-supervised training of our neural renderer on raw image collections. Our method represents a significant step in bringing the explicit physical control over lights available in typical 3D computer graphics tools, such as Blender, to in-the-wild relighting.

Problem

• Self-supervised image relighting with full control
• Combining physical rendering accuracy with neural realism
• Differentiable rendering for in-the-wild illumination reconstruction

Innovation

• Self-supervised neural rendering for relighting
• Combines physical rendering with neural networks
• Differentiable rendering for illumination reconstruction