Intrinsic Image Fusion for Multi-View 3D Material Reconstruction

2025-12-15

AI Summary

Multi-view material reconstruction suffers from poor physical consistency, severe under-constrained optimization, and heavy reliance on noisy path tracing. Method: We propose an end-to-end differentiable framework that jointly integrates single-view diffusion priors with multi-view geometric consistency constraints; introduces, for the first time, a soft multi-view confidence-weighted fusion mechanism; replaces black-box prediction with low-dimensional explicit parameterization; and employs inverse path tracing optimization to enforce cross-view material consistency without ground-truth supervision. Contribution/Results: Our method achieves significant improvements over state-of-the-art approaches on both synthetic and real-world datasets. It enables more accurate material disentanglement, yields sharper object boundaries, substantially reduces noise, and supports high-fidelity relighting rendering.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We introduce Intrinsic Image Fusion, a method that reconstructs high-quality physically based materials from multi-view images. Material reconstruction is highly underconstrained and typically relies on analysis-by-synthesis, which requires expensive and noisy path tracing. To better constrain the optimization, we incorporate single-view priors into the reconstruction process. We leverage a diffusion-based material estimator that produces multiple, but often inconsistent, candidate decompositions per view. To reduce the inconsistency, we fit an explicit low-dimensional parametric function to the predictions. We then propose a robust optimization framework using soft per-view prediction selection together with confidence-based soft multi-view inlier set to fuse the most consistent predictions of the most confident views into a consistent parametric material space. Finally, we use inverse path tracing to optimize for the low-dimensional parameters. Our results outperform state-of-the-art methods in material disentanglement on both synthetic and real scenes, producing sharp and clean reconstructions suitable for high-quality relighting.

Problem

• Reconstructs high-quality materials from multi-view images
• Reduces inconsistencies in single-view material predictions
• Optimizes low-dimensional parameters for sharp relighting reconstructions

Innovation

• Uses diffusion-based material estimator for single-view priors
• Fits explicit low-dimensional parametric function to predictions
• Optimizes with soft multi-view fusion and inverse path tracing