HoloPart: Generative 3D Part Amodal Segmentation

2025-04-10

AI Summary

This paper addresses the unsolved challenge of complete, semantically consistent part segmentation for occluded 3D shapes—introducing “3D part amodal segmentation” as a novel task. Methodologically, we propose a two-stage framework: first generating an initial part segmentation from visible regions, then reconstructing the full structure—including occluded parts—via a dual-attention diffusion model that jointly incorporates local geometric details and global shape consistency. Our technical contributions are threefold: (1) the first 3D diffusion architecture tailored for part-level completion, integrating local attention with global shape-context attention; (2) the construction of a new benchmark, ABO/PartObjaverse-Tiny, specifically designed for amodal part segmentation; and (3) state-of-the-art performance on this benchmark, significantly outperforming existing shape completion and segmentation methods. The resulting high-fidelity, semantically coherent part segmentations enable downstream applications such as geometric editing, rigging for animation, and material assignment.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

3D part amodal segmentation--decomposing a 3D shape into complete, semantically meaningful parts, even when occluded--is a challenging but crucial task for 3D content creation and understanding. Existing 3D part segmentation methods only identify visible surface patches, limiting their utility. Inspired by 2D amodal segmentation, we introduce this novel task to the 3D domain and propose a practical, two-stage approach, addressing the key challenges of inferring occluded 3D geometry, maintaining global shape consistency, and handling diverse shapes with limited training data. First, we leverage existing 3D part segmentation to obtain initial, incomplete part segments. Second, we introduce HoloPart, a novel diffusion-based model, to complete these segments into full 3D parts. HoloPart utilizes a specialized architecture with local attention to capture fine-grained part geometry and global shape context attention to ensure overall shape consistency. We introduce new benchmarks based on the ABO and PartObjaverse-Tiny datasets and demonstrate that HoloPart significantly outperforms state-of-the-art shape completion methods. By incorporating HoloPart with existing segmentation techniques, we achieve promising results on 3D part amodal segmentation, opening new avenues for applications in geometry editing, animation, and material assignment.

Problem

• Decompose 3D shapes into complete occluded parts
• Infer occluded 3D geometry with shape consistency
• Handle diverse shapes with limited training data

Innovation

• Two-stage approach for 3D part segmentation
• Diffusion-based model HoloPart completes occluded parts
• Combines local and global attention for shape consistency