Materialist: Physically Based Editing Using Single-Image Inverse Rendering

2025-01-07

AI Summary

This work addresses the challenge of photorealistic rendering—particularly illumination, shadowing, and transparency control—in single-image material editing. We propose the first single-image inverse physically based rendering framework, which does not require full scene geometry. Our method employs a neural network to predict initial material properties and integrates them with a progressive differentiable renderer grounded in the rendering equation, jointly optimizing environmental lighting and material parameters. The framework enables high-fidelity material editing, object insertion, relighting, and refractive modeling for transparent materials. Quantitatively and qualitatively, it outperforms state-of-the-art single-view approaches—including neural radiance fields (NeRF) and Stable Diffusion—across key metrics: photorealism, global illumination consistency, shadow accuracy, and physical interpretability of inferred parameters.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Data transparency and provenance

Abstract

To perform image editing based on single-view, inverse physically based rendering, we present a method combining a learning-based approach with progressive differentiable rendering. Given an image, our method leverages neural networks to predict initial material properties. Progressive differentiable rendering is then used to optimize the environment map and refine the material properties with the goal of closely matching the rendered result to the input image. We require only a single image while other inverse rendering methods based on the rendering equation require multiple views. In comparison to single-view methods that rely on neural renderers, our approach achieves more realistic light material interactions, accurate shadows, and global illumination. Furthermore, with optimized material properties and illumination, our method enables a variety of tasks, including physically based material editing, object insertion, and relighting. We also propose a method for material transparency editing that operates effectively without requiring full scene geometry. Compared with methods based on Stable Diffusion, our approach offers stronger interpretability and more realistic light refraction based on empirical results.

Problem

• Single Image Editing
• Material Rendering
• Transparency Adjustment

Innovation

• Single-image Material Editing
• Interactive Lighting Optimization
• Transparent Material Adjustment