Latent Intrinsics Emerge from Training to Relight

2024-05-31

Neural Information Processing Systems

AI Summary

Image relighting aims to synthesize the appearance of a source scene under novel lighting conditions. Traditional inverse graphics approaches rely on explicit geometric and material modeling, suffering from error accumulation and representational limitations. This paper proposes a fully data-driven implicit relighting framework that jointly models geometry, material properties (e.g., albedo), and lighting as a unified latent variable, eliminating explicit intermediate representations. Built upon a variational autoencoder architecture, it end-to-end learns lighting-invariant intrinsic representations and a differentiable rendering mapping. We theoretically and empirically demonstrate— for the first time—that the model spontaneously discovers semantically interpretable intrinsic decompositions under purely unsupervised training; remarkably, albedo recovery accuracy matches state-of-the-art supervised methods without any ground-truth albedo supervision. On real-scene relighting tasks and benchmarks including MIT-Adobe FiveK, our method achieves superior quantitative performance over existing unsupervised approaches and most supervised counterparts.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Image relighting is the task of showing what a scene from a source image would look like if illuminated differently. Inverse graphics schemes recover an explicit representation of geometry and a set of chosen intrinsics, then relight with some form of renderer. However error control for inverse graphics is difficult, and inverse graphics methods can represent only the effects of the chosen intrinsics. This paper describes a relighting method that is entirely data-driven, where intrinsics and lighting are each represented as latent variables. Our approach produces SOTA relightings of real scenes, as measured by standard metrics. We show that albedo can be recovered from our latent intrinsics without using any example albedos, and that the albedos recovered are competitive with SOTA methods.

Problem

• Develop data-driven image relighting with latent variables
• Recover albedo without using example albedos
• Achieve state-of-the-art relighting performance

Innovation

• Data-driven relighting with latent variables
• Recovers albedo without example albedos
• Competitive with SOTA relighting methods