MeshArt: Generating Articulated Meshes with Structure-guided Transformers

2024-12-16

arXiv.org

AI Summary

To address geometric ambiguity and poor part connectivity in 3D mesh generation for articulated objects, this paper proposes a two-stage hierarchical Transformer framework: the first stage generates joint-aware part topology, while the second stage autoregressively synthesizes triangular faces conditioned on this topology. Innovatively, both joint structures and mesh faces are unified as quantized triangular embeddings, and a structure-guided conditional mechanism is introduced to explicitly encode local connectivity and articulation patterns, ensuring global consistency. The method integrates structured sequence modeling, hierarchical autoregressive generation, and boundary primitive encoding. Experiments demonstrate a 57.1% improvement in structural coverage and a 209-point reduction in mesh Fréchet Inception Distance (FID), significantly outperforming state-of-the-art approaches.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Diffusion Models for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Articulated 3D object generation is fundamental for creating realistic, functional, and interactable virtual assets which are not simply static. We introduce MeshArt, a hierarchical transformer-based approach to generate articulated 3D meshes with clean, compact geometry, reminiscent of human-crafted 3D models. We approach articulated mesh generation in a part-by-part fashion across two stages. First, we generate a high-level articulation-aware object structure; then, based on this structural information, we synthesize each part's mesh faces. Key to our approach is modeling both articulation structures and part meshes as sequences of quantized triangle embeddings, leading to a unified hierarchical framework with transformers for autoregressive generation. Object part structures are first generated as their bounding primitives and articulation modes; a second transformer, guided by these articulation structures, then generates each part's mesh triangles. To ensure coherency among generated parts, we introduce structure-guided conditioning that also incorporates local part mesh connectivity. MeshArt shows significant improvements over state of the art, with 57.1% improvement in structure coverage and a 209-point improvement in mesh generation FID.

Problem

• Generating articulated 3D meshes with clean geometry
• Modeling articulation structures and part meshes as sequences
• Ensuring coherency among generated parts with structure-guided conditioning

Innovation

• Hierarchical transformer-based articulated mesh generation
• Quantized triangle embeddings for unified framework
• Structure-guided conditioning for part coherence