TexGaussian: Generating High-quality PBR Material via Octree-based 3D Gaussian Splatting

2024-11-29

arXiv.org

AI Summary

Existing methods for automatic PBR material generation in 3D content creation struggle to simultaneously ensure multi-view texture consistency and precise geometry-texture alignment. Method: This paper proposes an end-to-end differentiable regression framework that aligns 3D Gaussian splatting with octree-structured geometry—binding Gaussians to octree leaf nodes—to enable joint, differentiable rendering of albedo, roughness, and metallic channels. By bypassing diffusion-based synthesis, the method performs material prediction via a single forward pass, inherently guaranteeing geometric coherence and texture continuity. Contributions/Results: Evaluated on public benchmarks, our approach achieves significant improvements in material visual fidelity and mesh-texture alignment accuracy. It attains inference speeds several times faster than state-of-the-art methods, while supporting both unconditional generation and text-guided synthesis—demonstrating versatility and efficiency without compromising physical plausibility or spatial consistency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Physically Based Rendering (PBR) materials play a crucial role in modern graphics, enabling photorealistic rendering across diverse environment maps. Developing an effective and efficient algorithm that is capable of automatically generating high-quality PBR materials rather than RGB texture for 3D meshes can significantly streamline the 3D content creation. Most existing methods leverage pre-trained 2D diffusion models for multi-view image synthesis, which often leads to severe inconsistency between the generated textures and input 3D meshes. This paper presents TexGaussian, a novel method that uses octant-aligned 3D Gaussian Splatting for rapid PBR material generation. Specifically, we place each 3D Gaussian on the finest leaf node of the octree built from the input 3D mesh to render the multi-view images not only for the albedo map but also for roughness and metallic. Moreover, our model is trained in a regression manner instead of diffusion denoising, capable of generating the PBR material for a 3D mesh in a single feed-forward process. Extensive experiments on publicly available benchmarks demonstrate that our method synthesizes more visually pleasing PBR materials and runs faster than previous methods in both unconditional and text-conditional scenarios, exhibiting better consistency with the given geometry. Our code and trained models are available at https://3d-aigc.github.io/TexGaussian.

Problem

• Generates high-quality PBR materials for 3D meshes
• Resolves inconsistency between textures and 3D geometry
• Uses octree-based 3D Gaussian Splatting for efficiency

Innovation

• Octree-based 3D Gaussian Splatting for PBR
• Single feed-forward PBR material generation
• Regression training instead of diffusion