Extracting Neural Materials from Multi-view Images

2026-06-25

AI Summary

Accurately reconstructing neural materials that capture complex specular and diffuse effects from multi-view images remains challenging, as conventional inverse rendering methods struggle to optimize the nonlinear latent space of neural materials. This work proposes NeuMatEx, a differentiable inverse rendering framework that introduces a Large Material Reconstruction Model (LMRM) to directly predict base color, neural material latent variables, and their associated uncertainties. These predictions provide high-quality initial estimates and dynamic constraints for subsequent inverse path tracing optimization. By leveraging uncertainty-guided weighted optimization, NeuMatEx effectively prevents lighting and highlights from being erroneously baked into the material parameters, enabling more accurate material decomposition. Experiments demonstrate that NeuMatEx significantly outperforms existing physically based rendering (PBR)-based methods on both synthetic and real-world data, producing neural materials with superior visual fidelity and improved physical plausibility.

Abstract

Neural materials can represent complex specular reflections and scattering effects in a compact, universal basis. However, acquiring and authoring such materials remains challenging. We present NeuMatEx, a differentiable inverse rendering method for extracting spatially varying neural materials from images. The nonlinear structure of neural material latent spaces makes optimization with naive inverse rendering infeasible. To address this, we train a Large Material Reconstruction Model (LMRM) that directly predicts initialbase color, neural material latents, and aleatoric uncertainty guides from images. This material prior provides a good initialization and better constrains our subsequent optimization using inverse path tracing. The predicted uncertainty further helps by anchoring high-confidence regions more tightly to the LMRM prediction, preventing lighting and complex specular effects from being baked into materials. Experiments on synthetic and real assets show that NeuMatEx extracts complex materials with better visual quality and material decomposition than PBR-based methods.

Problem

• neural materials
• inverse rendering
• material extraction
• multi-view images
• spatially varying materials

Innovation

• neural materials
• inverse rendering
• differentiable rendering
• material decomposition
• uncertainty-guided optimization