OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesion

2025-07-08

AI Summary

Existing 3D generative methods predominantly produce monolithic shapes, limiting interactive editing capabilities. To address this, we propose a two-stage collaborative framework for part-level controllable 3D asset generation. In the first stage, unlabeled 3D part layout planning is guided by 2D part masks to ensure structural consistency. In the second stage, an autoregressive structural planning module generates a sequence of part bounding boxes, while a pretrained spatially conditioned correction flow model jointly synthesizes geometry and texture. Our method supports user-defined part granularity and precise spatial placement, enabling unified modeling of semantic disentanglement and structural coherence. Extensive evaluations on multiple benchmarks demonstrate state-of-the-art performance in both fidelity and diversity, with significant improvements in editability and downstream applicability—e.g., for scene composition and iterative design.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The creation of 3D assets with explicit, editable part structures is crucial for advancing interactive applications, yet most generative methods produce only monolithic shapes, limiting their utility. We introduce OmniPart, a novel framework for part-aware 3D object generation designed to achieve high semantic decoupling among components while maintaining robust structural cohesion. OmniPart uniquely decouples this complex task into two synergistic stages: (1) an autoregressive structure planning module generates a controllable, variable-length sequence of 3D part bounding boxes, critically guided by flexible 2D part masks that allow for intuitive control over part decomposition without requiring direct correspondences or semantic labels; and (2) a spatially-conditioned rectified flow model, efficiently adapted from a pre-trained holistic 3D generator, synthesizes all 3D parts simultaneously and consistently within the planned layout. Our approach supports user-defined part granularity, precise localization, and enables diverse downstream applications. Extensive experiments demonstrate that OmniPart achieves state-of-the-art performance, paving the way for more interpretable, editable, and versatile 3D content.

Problem

• Generates 3D objects with editable part structures
• Decouples semantic parts while ensuring structural cohesion
• Enables user-controlled part granularity and precise localization

Innovation

• Autoregressive module generates part bounding boxes
• Rectified flow model synthesizes 3D parts simultaneously
• Supports user-defined part granularity and localization