Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assets

2025-05-12

AI Summary

High-quality, controllable 3D asset generation remains hindered by data scarcity, limited algorithmic fidelity, and fragmented tooling ecosystems. This work introduces the first end-to-end open-source framework addressing these three core bottlenecks. Methodologically, it proposes a two-stage 3D-native architecture enabling direct adaptation of 2D control techniques (e.g., LoRA) to 3D generation; constructs the first standardized, high-fidelity 3D asset dataset comprising 2 million samples; and integrates a hybrid VAE-DiT geometric generator, diffusion-based texture synthesis, Perceiver-based latent encoding, TSDF geometry representation, and a geometry-texture co-diffusion mechanism. Experiments demonstrate state-of-the-art open-source performance in both geometry and texture quality—on par with leading closed-source systems—while supporting fine-grained, multi-modal conditioning (text, image, and control signals) and significantly improving cross-view consistency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

While generative artificial intelligence has advanced significantly across text, image, audio, and video domains, 3D generation remains comparatively underdeveloped due to fundamental challenges such as data scarcity, algorithmic limitations, and ecosystem fragmentation. To this end, we present Step1X-3D, an open framework addressing these challenges through: (1) a rigorous data curation pipeline processing>5M assets to create a 2M high-quality dataset with standardized geometric and textural properties; (2) a two-stage 3D-native architecture combining a hybrid VAE-DiT geometry generator with an diffusion-based texture synthesis module; and (3) the full open-source release of models, training code, and adaptation modules. For geometry generation, the hybrid VAE-DiT component produces TSDF representations by employing perceiver-based latent encoding with sharp edge sampling for detail preservation. The diffusion-based texture synthesis module then ensures cross-view consistency through geometric conditioning and latent-space synchronization. Benchmark results demonstrate state-of-the-art performance that exceeds existing open-source methods, while also achieving competitive quality with proprietary solutions. Notably, the framework uniquely bridges the 2D and 3D generation paradigms by supporting direct transfer of 2D control techniques~(e.g., LoRA) to 3D synthesis. By simultaneously advancing data quality, algorithmic fidelity, and reproducibility, Step1X-3D aims to establish new standards for open research in controllable 3D asset generation.

Problem

• Addressing data scarcity and quality in 3D asset generation
• Developing a hybrid VAE-DiT geometry and texture synthesis model
• Enhancing cross-view consistency and control in 3D generation

Innovation

• Hybrid VAE-DiT architecture for geometry generation
• Diffusion-based module for consistent texture synthesis
• Open-source framework with 2D-to-3D control transfer