ComPC: Completing a 3D Point Cloud with 2D Diffusion Priors

2024-04-10

AI Summary

To address the incompleteness of sensor-acquired 3D point clouds caused by self-occlusion and the poor generalizability of existing completion methods—due to their reliance on category-specific training data—this paper proposes the first training-free, test-time zero-shot point cloud completion framework for unseen categories. Our method leverages a pre-trained 2D diffusion model as a cross-modal prior to guide 3D generation; introduces Partial Gaussian Initialization for geometry-aware latent initialization; designs Zero-shot Fractal Completion to enhance structural consistency via fractal priors; and incorporates a differentiable Point Cloud Extraction module for end-to-end optimization. Extensive experiments on both synthetic and real-world scan datasets demonstrate significant improvements over state-of-the-art methods, achieving cross-category generalization, high-fidelity reconstruction, and uniformly distributed output points.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

3D point clouds directly collected from objects through sensors are often incomplete due to self-occlusion. Conventional methods for completing these partial point clouds rely on manually organized training sets and are usually limited to object categories seen during training. In this work, we propose a test-time framework for completing partial point clouds across unseen categories without any requirement for training. Leveraging point rendering via Gaussian Splatting, we develop techniques of Partial Gaussian Initialization, Zero-shot Fractal Completion, and Point Cloud Extraction that utilize priors from pre-trained 2D diffusion models to infer missing regions and extract uniform completed point clouds. Experimental results on both synthetic and real-world scanned point clouds demonstrate that our approach outperforms existing methods in completing a variety of objects. Our project page is at url{https://tianxinhuang.github.io/projects/ComPC/}.

Problem

• 3D Point Cloud Completion
• Object Occlusion
• Unseen Object Reconstruction

Innovation

• 3D Point Cloud Completion
• 2D Diffusion Prediction
• Gaussian Splatter Technique