MaterialMVP: Illumination-Invariant Material Generation via Multi-view PBR Diffusion

2025-03-13

AI Summary

To address key challenges in multi-view PBR material synthesis—including lighting sensitivity, inter-view inconsistency, and geometric misalignment—this paper proposes the first end-to-end diffusion-based framework that generates lighting-invariant and geometrically consistent material textures. Methodologically, we introduce Reference Attention to guide texture generation, design a dual-branch collaborative architecture with multi-channel alignment attention for precise spatial registration of albedo and metallic-roughness maps, and incorporate learnable material embeddings with consistency-aware regularization to enforce structural constraints across viewpoints and lighting conditions. Experiments demonstrate significant improvements over state-of-the-art methods in material photorealism, cross-view stability, and geometric fidelity. Our approach provides an efficient, robust solution for industrial-grade 3D asset generation, enabling high-quality, lighting-agnostic PBR texture synthesis.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Sustainability and carbon-aware systems for Web, mobile, and WoT
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Physically-based rendering (PBR) has become a cornerstone in modern computer graphics, enabling realistic material representation and lighting interactions in 3D scenes. In this paper, we present MaterialMVP, a novel end-to-end model for generating PBR textures from 3D meshes and image prompts, addressing key challenges in multi-view material synthesis. Our approach leverages Reference Attention to extract and encode informative latent from the input reference images, enabling intuitive and controllable texture generation. We also introduce a Consistency-Regularized Training strategy to enforce stability across varying viewpoints and illumination conditions, ensuring illumination-invariant and geometrically consistent results. Additionally, we propose Dual-Channel Material Generation, which separately optimizes albedo and metallic-roughness (MR) textures while maintaining precise spatial alignment with the input images through Multi-Channel Aligned Attention. Learnable material embeddings are further integrated to capture the distinct properties of albedo and MR. Experimental results demonstrate that our model generates PBR textures with realistic behavior across diverse lighting scenarios, outperforming existing methods in both consistency and quality for scalable 3D asset creation.

Problem

• Generates PBR textures from 3D meshes and image prompts.
• Ensures illumination-invariant and geometrically consistent material synthesis.
• Optimizes albedo and metallic-roughness textures with precise spatial alignment.

Innovation

• Reference Attention for latent encoding
• Consistency-Regularized Training for stability
• Dual-Channel Material Generation optimization