Repurposing 2D Diffusion Models for 3D Shape Completion

2025-12-15

AI Summary

Addressing the scarcity of high-quality 3D data and the modality gap between 3D point clouds and the latent space of pre-trained 2D diffusion models, this paper introduces Shape Atlas—a compact, differentiable 2D geometric representation—enabling the first successful adaptation of pre-trained 2D diffusion models to 3D point cloud completion. Our method comprises three key components: (1) encoding incomplete point clouds into Shape Atlas representations; (2) lightweight fine-tuning of a 2D diffusion model to incorporate geometric priors; and (3) a conditional feature alignment mechanism ensuring cross-modal generation consistency. Evaluated on PCN and ShapeNet-55, our approach achieves state-of-the-art completion accuracy, reducing Chamfer Distance by 18.7% and improving F-Score by 5.2%. Moreover, it supports end-to-end generation of high-fidelity, topologically plausible mesh models, demonstrating practical utility in real-world 3D content creation scenarios.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present a framework that adapts 2D diffusion models for 3D shape completion from incomplete point clouds. While text-to-image diffusion models have achieved remarkable success with abundant 2D data, 3D diffusion models lag due to the scarcity of high-quality 3D datasets and a persistent modality gap between 3D inputs and 2D latent spaces. To overcome these limitations, we introduce the Shape Atlas, a compact 2D representation of 3D geometry that (1) enables full utilization of the generative power of pretrained 2D diffusion models, and (2) aligns the modalities between the conditional input and output spaces, allowing more effective conditioning. This unified 2D formulation facilitates learning from limited 3D data and produces high-quality, detail-preserving shape completions. We validate the effectiveness of our results on the PCN and ShapeNet-55 datasets. Additionally, we show the downstream application of creating artist-created meshes from our completed point clouds, further demonstrating the practicality of our method.

Problem

• Adapts 2D diffusion models for 3D shape completion
• Overcomes 3D data scarcity and modality gap issues
• Enables high-quality completion from incomplete point clouds

Innovation

• Adapts 2D diffusion models for 3D shape completion
• Introduces Shape Atlas for 2D representation of 3D geometry
• Enables effective conditioning with limited 3D data