SeaLion: Semantic Part-Aware Latent Point Diffusion Models for 3D Generation

2025-05-23

AI Summary

Existing point cloud generation models largely neglect fine-grained part-segmentation-aware synthesis and lack appropriate evaluation metrics for part-level fidelity. Method: We propose a semantic-part-aware latent point diffusion framework that jointly predicts latent point noise and part labels during denoising, enabling semantic-controllable reconstruction via conditional latent decoding. We introduce the part-aware Chamfer Distance (p-CD) as a novel metric and establish the first 1-NNA (1-Nearest Neighbor Accuracy) evaluation framework supporting part-level quality assessment. Our method is the first to enable end-to-end joint modeling of geometry and semantic part labels within diffusion models and supports semi-supervised training to reduce annotation cost. Results: On ShapeNet and IntrA, our 1-NNA (p-CD) scores surpass DiffFacto by 13.33% and 6.52%, respectively, demonstrating superior performance in generative data augmentation and part-level 3D editing.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Denoising diffusion probabilistic models have achieved significant success in point cloud generation, enabling numerous downstream applications, such as generative data augmentation and 3D model editing. However, little attention has been given to generating point clouds with point-wise segmentation labels, as well as to developing evaluation metrics for this task. Therefore, in this paper, we present SeaLion, a novel diffusion model designed to generate high-quality and diverse point clouds with fine-grained segmentation labels. Specifically, we introduce the semantic part-aware latent point diffusion technique, which leverages the intermediate features of the generative models to jointly predict the noise for perturbed latent points and associated part segmentation labels during the denoising process, and subsequently decodes the latent points to point clouds conditioned on part segmentation labels. To effectively evaluate the quality of generated point clouds, we introduce a novel point cloud pairwise distance calculation method named part-aware Chamfer distance (p-CD). This method enables existing metrics, such as 1-NNA, to measure both the local structural quality and inter-part coherence of generated point clouds. Experiments on the large-scale synthetic dataset ShapeNet and real-world medical dataset IntrA demonstrate that SeaLion achieves remarkable performance in generation quality and diversity, outperforming the existing state-of-the-art model, DiffFacto, by 13.33% and 6.52% on 1-NNA (p-CD) across the two datasets. Experimental analysis shows that SeaLion can be trained semi-supervised, thereby reducing the demand for labeling efforts. Lastly, we validate the applicability of SeaLion in generative data augmentation for training segmentation models and the capability of SeaLion to serve as a tool for part-aware 3D shape editing.

Problem

• Generates point clouds with fine-grained segmentation labels
• Introduces part-aware Chamfer distance for evaluation
• Enables semi-supervised training to reduce labeling efforts

Innovation

• Semantic part-aware latent point diffusion technique
• Part-aware Chamfer distance (p-CD) metric
• Semi-supervised training for reduced labeling