ROGR: Relightable 3D Objects using Generative Relighting

2025-10-03

AI Summary

This work addresses multi-view image-driven relightable 3D object reconstruction for photorealistic appearance synthesis under arbitrary environmental illumination. To this end, we propose a dual-branch neural radiance field (NeRF) architecture that explicitly decouples global diffuse and specular reflectance components, circumventing per-light optimization or complex light transport simulation employed by conventional methods. We introduce an illumination-condition control mechanism and a multi-illumination appearance sampling strategy, enabling feed-forward, single-pass relighting from environment maps. Our method achieves state-of-the-art performance on the TensoIR and Stanford-ORB datasets and demonstrates strong generalization and practicality on real-world object reconstruction. The core contribution is the first end-to-end differentiable, generative NeRF framework for relighting that requires no illumination-specific fine-tuning.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We introduce ROGR, a novel approach that reconstructs a relightable 3D model of an object captured from multiple views, driven by a generative relighting model that simulates the effects of placing the object under novel environment illuminations. Our method samples the appearance of the object under multiple lighting environments, creating a dataset that is used to train a lighting-conditioned Neural Radiance Field (NeRF) that outputs the object's appearance under any input environmental lighting. The lighting-conditioned NeRF uses a novel dual-branch architecture to encode the general lighting effects and specularities separately. The optimized lighting-conditioned NeRF enables efficient feed-forward relighting under arbitrary environment maps without requiring per-illumination optimization or light transport simulation. We evaluate our approach on the established TensoIR and Stanford-ORB datasets, where it improves upon the state-of-the-art on most metrics, and showcase our approach on real-world object captures.

Problem

• Reconstructs relightable 3D models from multi-view captures
• Trains lighting-conditioned NeRF for arbitrary environment illumination
• Enables efficient feed-forward relighting without per-illumination optimization

Innovation

• Uses lighting-conditioned NeRF for object appearance modeling
• Employs dual-branch architecture to separate lighting effects
• Enables feed-forward relighting without per-illumination optimization