Advancing high-fidelity 3D and Texture Generation with 2.5D latents

2025-05-27

AI Summary

Existing 3D generation methods suffer from structural-textural misalignment and limited fidelity due to stage-wise, heterogeneous modeling of geometry and texture. To address this, we propose a unified generative framework based on a bidirectionally invertible 2.5D latent space: multi-view RGB, normal, and coordinate maps are jointly embedded into a shared latent representation, enabling end-to-end, text- or image-conditioned 3D synthesis. We introduce the first 2.5D latent variable formulation, uniquely supporting joint optimization of geometry and appearance. Furthermore, we design a lightweight 2.5D-to-3D refinement decoder that significantly improves texture consistency under geometric guidance. Our method achieves state-of-the-art performance across text-to-3D and image-to-3D benchmarks, yielding a +4.2 dB PSNR gain in geometry-guided texture reconstruction and markedly enhanced structural-color consistency.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Despite the availability of large-scale 3D datasets and advancements in 3D generative models, the complexity and uneven quality of 3D geometry and texture data continue to hinder the performance of 3D generation techniques. In most existing approaches, 3D geometry and texture are generated in separate stages using different models and non-unified representations, frequently leading to unsatisfactory coherence between geometry and texture. To address these challenges, we propose a novel framework for joint generation of 3D geometry and texture. Specifically, we focus in generate a versatile 2.5D representations that can be seamlessly transformed between 2D and 3D. Our approach begins by integrating multiview RGB, normal, and coordinate images into a unified representation, termed as 2.5D latents. Next, we adapt pre-trained 2D foundation models for high-fidelity 2.5D generation, utilizing both text and image conditions. Finally, we introduce a lightweight 2.5D-to-3D refiner-decoder framework that efficiently generates detailed 3D representations from 2.5D images. Extensive experiments demonstrate that our model not only excels in generating high-quality 3D objects with coherent structure and color from text and image inputs but also significantly outperforms existing methods in geometry-conditioned texture generation.

Problem

• Addressing incoherence between 3D geometry and texture generation
• Overcoming limitations of separate 3D and texture modeling approaches
• Enhancing fidelity in 2D-to-3D conversion using unified 2.5D representations

Innovation

• Joint 3D geometry and texture generation framework
• Versatile 2.5D latent representations for transformation
• Lightweight 2.5D-to-3D refiner-decoder for detail