MARS: Mesh AutoRegressive Model for 3D Shape Detailization

2025-02-17

AI Summary

Existing mesh detailing methods predominantly rely on category-specific GANs, suffering from poor cross-category generalization and insufficient multi-level geometric consistency constraints. To address this, we propose the first unified autoregressive mesh modeling framework supporting multiple categories and multiple Levels of Detail (LOD). Our method leverages LOD sequence modeling to enable cross-category geometric supervision, incorporates discrete mesh tokenization with a Transformer architecture, and introduces a latent-space geometric alignment loss to preserve shape integrity. By abandoning the conventional GAN paradigm, our approach achieves state-of-the-art performance on 3D detailing benchmarks—significantly improving both fine-grained detail fidelity and global structural coherence. Notably, it is the first method to support zero-shot cross-category detail generation, enabling high-fidelity geometry synthesis across unseen categories without retraining.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

State-of-the-art methods for mesh detailization predominantly utilize Generative Adversarial Networks (GANs) to generate detailed meshes from coarse ones. These methods typically learn a specific style code for each category or similar categories without enforcing geometry supervision across different Levels of Detail (LODs). Consequently, such methods often fail to generalize across a broader range of categories and cannot ensure shape consistency throughout the detailization process. In this paper, we introduce MARS, a novel approach for 3D shape detailization. Our method capitalizes on a novel multi-LOD, multi-category mesh representation to learn shape-consistent mesh representations in latent space across different LODs. We further propose a mesh autoregressive model capable of generating such latent representations through next-LOD token prediction. This approach significantly enhances the realism of the generated shapes. Extensive experiments conducted on the challenging 3D Shape Detailization benchmark demonstrate that our proposed MARS model achieves state-of-the-art performance, surpassing existing methods in both qualitative and quantitative assessments. Notably, the model's capability to generate fine-grained details while preserving the overall shape integrity is particularly commendable.

Problem

• Improves 3D mesh detailization across categories
• Ensures shape consistency across Levels of Detail
• Enhances realism in generated 3D shapes

Innovation

• Multi-LOD mesh representation
• Mesh autoregressive model
• Shape-consistent latent space