A3D: Does Diffusion Dream about 3D Alignment?

2024-06-21

arXiv.org

AI Summary

This work addresses the challenge of cross-object geometric alignment of semantic parts in text-driven 3D generation: existing methods generate objects independently, resulting in inconsistent poses and structures—failing to meet the demand for coherent modeling in 3D asset design. To this end, we propose, for the first time, joint multi-object generation within a shared latent space, where semantic transitions are modeled via continuous latent-space interpolation, jointly regularized for transition smoothness and intermediate shape plausibility. Our method builds upon the Score Distillation Sampling (SDS) framework, integrating geometric consistency regularization with differentiable rendering. It achieves high-fidelity, cross-object-aligned 3D generation across multiple categories, significantly improving 3D editing and object compositing quality. Code and interactive demos are publicly available.

Technology Category

• Natural Language Processing: Generation
• Computer Vision: 3D Computer Vision
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

We tackle the problem of text-driven 3D generation from a geometry alignment perspective. Given a set of text prompts, we aim to generate a collection of objects with semantically corresponding parts aligned across them. Recent methods based on Score Distillation have succeeded in distilling the knowledge from 2D diffusion models to high-quality representations of the 3D objects. These methods handle multiple text queries separately, and therefore the resulting objects have a high variability in object pose and structure. However, in some applications, such as 3D asset design, it may be desirable to obtain a set of objects aligned with each other. In order to achieve the alignment of the corresponding parts of the generated objects, we propose to embed these objects into a common latent space and optimize the continuous transitions between these objects. We enforce two kinds of properties of these transitions: smoothness of the transition and plausibility of the intermediate objects along the transition. We demonstrate that both of these properties are essential for good alignment. We provide several practical scenarios that benefit from alignment between the objects, including 3D editing and object hybridization, and experimentally demonstrate the effectiveness of our method. https://voyleg.github.io/a3d/

Problem

• Generate 3D objects with aligned parts from text prompts.
• Embed objects in a common latent space for alignment.
• Ensure smooth and plausible transitions between aligned objects.

Innovation

• Embed objects into common latent space
• Optimize smooth, plausible transitions between objects
• Align 3D objects using text-driven generation