MultiDreamer3D: Multi-concept 3D Customization with Concept-Aware Diffusion Guidance

2025-01-23

AI Summary

This work addresses the challenging problem of 3D multi-concept co-customization—specifically, cross-concept interactions and dynamic attribute modification. We propose the first end-to-end framework: (1) leveraging large language models to generate semantically aligned 3D layouts; (2) constructing fine-grained concept-labeled point clouds and initializing them as concept-annotated 3D Gaussians; and (3) introducing a concept-aware interval score matching mechanism integrated with diffusion priors for controllable optimization. Key contributions include: (1) a concept-label-guided initialization strategy for 3D Gaussians, and (2) a concept-aware score matching scheme enabling attribute editing and interaction modeling. Experiments demonstrate significant improvements over state-of-the-art baselines in multi-object existence, concept identity fidelity, and robustness under complex conditions—including occlusion and material variation—achieving high-quality, editable, and semantically precise multi-concept 3D generation.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

While single-concept customization has been studied in 3D, multi-concept customization remains largely unexplored. To address this, we propose MultiDreamer3D that can generate coherent multi-concept 3D content in a divide-and-conquer manner. First, we generate 3D bounding boxes using an LLM-based layout controller. Next, a selective point cloud generator creates coarse point clouds for each concept. These point clouds are placed in the 3D bounding boxes and initialized into 3D Gaussian Splatting with concept labels, enabling precise identification of concept attributions in 2D projections. Finally, we refine 3D Gaussians via concept-aware interval score matching, guided by concept-aware diffusion. Our experimental results show that MultiDreamer3D not only ensures object presence and preserves the distinct identities of each concept but also successfully handles complex cases such as property change or interaction. To the best of our knowledge, we are the first to address the multi-concept customization in 3D.

Problem

• Multi-concept Customization
• 3D World
• Concept Interaction

Innovation

• MultiDreamer3D
• Multi-concept Customization
• 3D Environment