NaTex: Seamless Texture Generation as Latent Color Diffusion

2025-11-20

AI Summary

Existing methods rely on multi-view diffusion models to generate 2D textures, which are then baked onto 3D meshes—suffering from poor occluded-region inpainting, misalignment between mesh geometry and texture boundaries, and inter-view inconsistencies in color and content. This work introduces the first native 3D texture generation framework: it represents texture as a dense colored point cloud and directly predicts surface colors in 3D space via an implicit color diffusion model. We propose a geometric-aware VAE coupled with a multi-control diffusion Transformer (DiT), jointly encoding geometry and spatial position to enable fine-grained geometry-texture alignment. End-to-end trained on 3D data, our method significantly outperforms state-of-the-art approaches in texture consistency and mesh fidelity. Moreover, the model exhibits strong generalization: without fine-tuning, it seamlessly transfers to downstream tasks including material synthesis, texture refinement, and part-aware semantic coloring.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We present NaTex, a native texture generation framework that predicts texture color directly in 3D space. In contrast to previous approaches that rely on baking 2D multi-view images synthesized by geometry-conditioned Multi-View Diffusion models (MVDs), NaTex avoids several inherent limitations of the MVD pipeline. These include difficulties in handling occluded regions that require inpainting, achieving precise mesh-texture alignment along boundaries, and maintaining cross-view consistency and coherence in both content and color intensity. NaTex features a novel paradigm that addresses the aforementioned issues by viewing texture as a dense color point cloud. Driven by this idea, we propose latent color diffusion, which comprises a geometry-awared color point cloud VAE and a multi-control diffusion transformer (DiT), entirely trained from scratch using 3D data, for texture reconstruction and generation. To enable precise alignment, we introduce native geometry control that conditions the DiT on direct 3D spatial information via positional embeddings and geometry latents. We co-design the VAE-DiT architecture, where the geometry latents are extracted via a dedicated geometry branch tightly coupled with the color VAE, providing fine-grained surface guidance that maintains strong correspondence with the texture. With these designs, NaTex demonstrates strong performance, significantly outperforming previous methods in texture coherence and alignment. Moreover, NaTex also exhibits strong generalization capabilities, either training-free or with simple tuning, for various downstream applications, e.g., material generation, texture refinement, and part segmentation and texturing.

Problem

• Generates 3D textures directly in 3D space
• Resolves occlusion and alignment issues in texture mapping
• Ensures cross-view consistency in texture content and color

Innovation

• Generates texture directly in 3D space
• Uses latent color diffusion with geometry-aware VAE
• Employs multi-control diffusion transformer for alignment