Large Material Gaussian Model for Relightable 3D Generation

2025-09-26

AI Summary

Existing 3D generative models struggle to synthesize physically plausible materials, resulting in static lighting and poor relighting capability. To address this, we propose the first large-scale 3D Gaussian representation method explicitly modeling PBR materials—jointly generating albedo, roughness, and metallic maps. Our approach leverages a multi-view diffusion model conditioned on depth and normal maps as geometric priors, employs a scalable Transformer architecture to enforce material-geometry alignment, and adopts a 3D Gaussian splatting representation for efficient yet high-fidelity rendering. Experiments demonstrate significant improvements over state-of-the-art baselines in material fidelity, cross-illumination generalization, and dynamic relighting quality. The method establishes a new paradigm for industrial-grade, automated 3D asset generation with photorealistic material properties.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

The increasing demand for 3D assets across various industries necessitates efficient and automated methods for 3D content creation. Leveraging 3D Gaussian Splatting, recent large reconstruction models (LRMs) have demonstrated the ability to efficiently achieve high-quality 3D rendering by integrating multiview diffusion for generation and scalable transformers for reconstruction. However, existing models fail to produce the material properties of assets, which is crucial for realistic rendering in diverse lighting environments. In this paper, we introduce the Large Material Gaussian Model (MGM), a novel framework designed to generate high-quality 3D content with Physically Based Rendering (PBR) materials, ie, albedo, roughness, and metallic properties, rather than merely producing RGB textures with uncontrolled light baking. Specifically, we first fine-tune a new multiview material diffusion model conditioned on input depth and normal maps. Utilizing the generated multiview PBR images, we explore a Gaussian material representation that not only aligns with 2D Gaussian Splatting but also models each channel of the PBR materials. The reconstructed point clouds can then be rendered to acquire PBR attributes, enabling dynamic relighting by applying various ambient light maps. Extensive experiments demonstrate that the materials produced by our method not only exhibit greater visual appeal compared to baseline methods but also enhance material modeling, thereby enabling practical downstream rendering applications.

Problem

• Generates 3D content with physically accurate material properties
• Enables dynamic relighting under diverse lighting environments
• Models PBR materials like albedo, roughness, and metallic

Innovation

• Fine-tunes multiview material diffusion model
• Explores Gaussian material representation for PBR
• Enables dynamic relighting with ambient light maps