3D-WAG: Hierarchical Wavelet-Guided Autoregressive Generation for High-Fidelity 3D Shapes

2024-11-28

arXiv.org

AI Summary

Traditional autoregressive models for large-scale 3D shape modeling suffer from high computational cost and low geometric fidelity due to voxel- or point-level “next-token” prediction. This work proposes the Wavelet Token Graph framework, the first to deeply integrate multi-scale wavelet analysis with Transformer-based autoregressive modeling. Instead of element-wise prediction, it reformulates generation as hierarchical “next-scale” prediction, enabling unconditional, class-conditional, and text-driven implicit distance field synthesis. Leveraging wavelet encoding, the method achieves efficient multi-scale tokenization—substantially shortening sequence length while preserving fine-grained geometric reconstruction capability. Experiments demonstrate state-of-the-art performance: superior coverage and MMD scores, higher-quality geometry, better distribution matching, and significantly accelerated inference.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: 3D Computer Vision
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Autoregressive (AR) models have achieved remarkable success in natural language and image generation, but their application to 3D shape modeling remains largely unexplored. Unlike diffusion models, AR models enable more efficient and controllable generation with faster inference times, making them especially suitable for data-intensive domains. Traditional 3D generative models using AR approaches often rely on ``next-token"predictions at the voxel or point level. While effective for certain applications, these methods can be restrictive and computationally expensive when dealing with large-scale 3D data. To tackle these challenges, we introduce 3D-WAG, an AR model for 3D implicit distance fields that can perform unconditional shape generation, class-conditioned and also text-conditioned shape generation. Our key idea is to encode shapes as multi-scale wavelet token maps and use a Transformer to predict the ``next higher-resolution token map"in an autoregressive manner. By redefining 3D AR generation task as ``next-scale"prediction, we reduce the computational cost of generation compared to traditional ``next-token"prediction models, while preserving essential geometric details of 3D shapes in a more structured and hierarchical manner. We evaluate 3D-WAG to showcase its benefit by quantitative and qualitative comparisons with state-of-the-art methods on widely used benchmarks. Our results show 3D-WAG achieves superior performance in key metrics like Coverage and MMD, generating high-fidelity 3D shapes that closely match the real data distribution.

Problem

• Efficient 3D shape generation using autoregressive models
• Reducing computational cost in large-scale 3D data processing
• Preserving geometric details in hierarchical wavelet-based generation

Innovation

• Hierarchical wavelet token maps encoding
• Next-scale prediction with Transformer
• Multi-scale autoregressive 3D generation