PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models

2024-12-24

AI Summary

Existing 3D generation and inpainting methods typically produce monolithic, unstructured models, lacking explicit semantic part representations and thus hindering fine-grained, semantics-aware editing. Method: We propose the first end-to-end 3D generation and inpainting framework explicitly designed for part-level semantic editability. Our approach employs a two-stage multi-view diffusion model that jointly achieves cross-view-consistent part segmentation and semantically plausible completion of occluded regions, subsequently driving implicit and explicit 3D reconstruction. Crucially, we pioneer the integration of multi-view diffusion into part-aware modeling, incorporating part-aware segmentation, occlusion-aware view completion, and context-aware part integration. Results: Our method significantly outperforms state-of-the-art part segmentation baselines on both synthetic and real-world data, enabling high-fidelity part replacement, recombination, and other downstream semantic editing tasks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Text- or image-to-3D generators and 3D scanners can now produce 3D assets with high-quality shapes and textures. These assets typically consist of a single, fused representation, like an implicit neural field, a Gaussian mixture, or a mesh, without any useful structure. However, most applications and creative workflows require assets to be made of several meaningful parts that can be manipulated independently. To address this gap, we introduce PartGen, a novel approach that generates 3D objects composed of meaningful parts starting from text, an image, or an unstructured 3D object. First, given multiple views of a 3D object, generated or rendered, a multi-view diffusion model extracts a set of plausible and view-consistent part segmentations, dividing the object into parts. Then, a second multi-view diffusion model takes each part separately, fills in the occlusions, and uses those completed views for 3D reconstruction by feeding them to a 3D reconstruction network. This completion process considers the context of the entire object to ensure that the parts integrate cohesively. The generative completion model can make up for the information missing due to occlusions; in extreme cases, it can hallucinate entirely invisible parts based on the input 3D asset. We evaluate our method on generated and real 3D assets and show that it outperforms segmentation and part-extraction baselines by a large margin. We also showcase downstream applications such as 3D part editing.

Problem

• 3D model generation
• 3D model repair
• modular components

Innovation

• PartGen
• 3D model decomposition
• reconstruction accuracy