3D Point Cloud Generation via Autoregressive Up-sampling

2025-03-11

AI Summary

To address the quality and efficiency bottlenecks in 3D point cloud generation caused by inherent disorder and structural irregularity, this paper proposes PointARU—a novel autoregressive upsampling framework that models point cloud generation as a coarse-to-fine hierarchical reconstruction process. Methodologically, PointARU innovatively integrates multi-scale discrete representation learning, 3D absolute positional encoding conditioned on decoded point clouds, and a lightweight, point-cloud-specific upsampling Transformer module. Compared to state-of-the-art diffusion-based approaches, PointARU achieves superior performance with significantly fewer parameters: it improves both Fréchet Inception Distance (FID) and Jensen–Shannon Divergence (JSD) metrics across benchmarks. Moreover, it establishes new state-of-the-art results on partial shape completion and sparse point cloud upsampling tasks, demonstrating strong generalization capability and practical applicability.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We introduce a pioneering autoregressive generative model for 3D point cloud generation. Inspired by visual autoregressive modeling (VAR), we conceptualize point cloud generation as an autoregressive up-sampling process. This leads to our novel model, PointARU, which progressively refines 3D point clouds from coarse to fine scales. PointARU follows a two-stage training paradigm: first, it learns multi-scale discrete representations of point clouds, and then it trains an autoregressive transformer for next-scale prediction. To address the inherent unordered and irregular structure of point clouds, we incorporate specialized point-based up-sampling network modules in both stages and integrate 3D absolute positional encoding based on the decoded point cloud at each scale during the second stage. Our model surpasses state-of-the-art (SoTA) diffusion-based approaches in both generation quality and parameter efficiency across diverse experimental settings, marking a new milestone for autoregressive methods in 3D point cloud generation. Furthermore, PointARU demonstrates exceptional performance in completing partial 3D shapes and up-sampling sparse point clouds, outperforming existing generative models in these tasks.

Problem

• Autoregressive model for 3D point cloud generation.
• Progressive refinement from coarse to fine scales.
• Superior performance in shape completion and up-sampling.

Innovation

• Autoregressive model for 3D point clouds
• Two-stage training with multi-scale representations
• Point-based up-sampling with positional encoding