MeshCraft: Exploring Efficient and Controllable Mesh Generation with Flow-based DiTs

2025-03-29

AI Summary

Existing AI-based 3D mesh generation suffers from low topological efficiency and uncontrollable face counts. Method: This paper introduces the first face-level triangular mesh streaming diffusion Transformer (DiT) framework. It pioneers streaming diffusion for face-level generation, designs a synergistic architecture combining a face-level VAE and a conditional diffusion Transformer, and enables non-autoregressive generation with continuous-space diffusion and precise face-count control. Contributions/Results: Our method generates 800-face meshes in just 3.2 seconds—35× faster than state-of-the-art methods—and achieves superior performance on ShapeNet and Objaverse. It supports multimodal conditioning—including text and sketch inputs—significantly reducing manual modeling effort. This work establishes a new paradigm for efficient, controllable, and high-quality 3D content generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In the domain of 3D content creation, achieving optimal mesh topology through AI models has long been a pursuit for 3D artists. Previous methods, such as MeshGPT, have explored the generation of ready-to-use 3D objects via mesh auto-regressive techniques. While these methods produce visually impressive results, their reliance on token-by-token predictions in the auto-regressive process leads to several significant limitations. These include extremely slow generation speeds and an uncontrollable number of mesh faces. In this paper, we introduce MeshCraft, a novel framework for efficient and controllable mesh generation, which leverages continuous spatial diffusion to generate discrete triangle faces. Specifically, MeshCraft consists of two core components: 1) a transformer-based VAE that encodes raw meshes into continuous face-level tokens and decodes them back to the original meshes, and 2) a flow-based diffusion transformer conditioned on the number of faces, enabling the generation of high-quality 3D meshes with a predefined number of faces. By utilizing the diffusion model for the simultaneous generation of the entire mesh topology, MeshCraft achieves high-fidelity mesh generation at significantly faster speeds compared to auto-regressive methods. Specifically, MeshCraft can generate an 800-face mesh in just 3.2 seconds (35$ imes$ faster than existing baselines). Extensive experiments demonstrate that MeshCraft outperforms state-of-the-art techniques in both qualitative and quantitative evaluations on ShapeNet dataset and demonstrates superior performance on Objaverse dataset. Moreover, it integrates seamlessly with existing conditional guidance strategies, showcasing its potential to relieve artists from the time-consuming manual work involved in mesh creation.

Problem

• Achieving efficient AI-generated 3D mesh topology
• Overcoming slow auto-regressive mesh generation speeds
• Enabling controllable mesh face count in generation

Innovation

• Flow-based diffusion transformer for mesh generation
• Transformer-based VAE for continuous face encoding
• Simultaneous mesh topology generation via diffusion