DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learning

2025-03-19

AI Summary

Existing autoregressive methods for 3D triangular mesh generation suffer from inherent limitations in maximum face count and topological incompleteness, compromising both geometric fidelity and fine-grained detail. To address this, we propose the first framework tailored for high-quality autoregressive triangular mesh generation. Our approach (1) introduces Direct Preference Optimization (DPO) to 3D generation—novelly integrating human aesthetic judgments with geometric metrics to construct preference pairs; (2) designs a vertex-aware triangular mesh tokenization strategy coupled with an efficient pretraining paradigm; and (3) enables dual-modality conditional modeling from both point clouds and images. Extensive experiments demonstrate that our method achieves state-of-the-art performance across topological correctness, geometric accuracy, and visual quality, significantly alleviating face-count constraints and structural incompleteness.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Triangle meshes play a crucial role in 3D applications for efficient manipulation and rendering. While auto-regressive methods generate structured meshes by predicting discrete vertex tokens, they are often constrained by limited face counts and mesh incompleteness. To address these challenges, we propose DeepMesh, a framework that optimizes mesh generation through two key innovations: (1) an efficient pre-training strategy incorporating a novel tokenization algorithm, along with improvements in data curation and processing, and (2) the introduction of Reinforcement Learning (RL) into 3D mesh generation to achieve human preference alignment via Direct Preference Optimization (DPO). We design a scoring standard that combines human evaluation with 3D metrics to collect preference pairs for DPO, ensuring both visual appeal and geometric accuracy. Conditioned on point clouds and images, DeepMesh generates meshes with intricate details and precise topology, outperforming state-of-the-art methods in both precision and quality. Project page: https://zhaorw02.github.io/DeepMesh/

Problem

• Overcome limitations in auto-regressive mesh generation methods.
• Improve mesh quality and detail through reinforcement learning.
• Align mesh generation with human preferences for visual appeal.

Innovation

• Efficient pre-training with novel tokenization algorithm
• Reinforcement Learning for human preference alignment
• Combines human evaluation with 3D metrics