Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generation

2024-12-30

AI Summary

This work addresses key challenges in text-to-3D scene generation—low geometric fidelity, slow inference, and weak text-3D alignment—by proposing a feed-forward, pixel-aligned multi-view 3D Gaussian voxel generation framework. Methodologically, it introduces (1) an RGB-D latent-space disentanglement scheme that explicitly separates geometry (depth) from appearance (color); (2) lightweight adaptation of a frozen text-to-image diffusion model, enabling unified object- and scene-level generation; and (3) synergistic integration of 3D Gaussian representations, latent diffusion priors, and multi-view joint training. The approach achieves end-to-end text-to-3D generation in seconds with millisecond-scale inference latency, significantly improving both structural accuracy and text–3D semantic alignment. It establishes a new paradigm for high-fidelity, scalable, feed-forward 3D reconstruction and synthesis.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

In this work, we introduce Prometheus, a 3D-aware latent diffusion model for text-to-3D generation at both object and scene levels in seconds. We formulate 3D scene generation as multi-view, feed-forward, pixel-aligned 3D Gaussian generation within the latent diffusion paradigm. To ensure generalizability, we build our model upon pre-trained text-to-image generation model with only minimal adjustments, and further train it using a large number of images from both single-view and multi-view datasets. Furthermore, we introduce an RGB-D latent space into 3D Gaussian generation to disentangle appearance and geometry information, enabling efficient feed-forward generation of 3D Gaussians with better fidelity and geometry. Extensive experimental results demonstrate the effectiveness of our method in both feed-forward 3D Gaussian reconstruction and text-to-3D generation. Project page: https://freemty.github.io/project-prometheus/

Problem

• 3D Scene Generation
• Text-to-3D
• Model Optimization

Innovation

• Prometheus model
• multi-view image generation
• separate color and shape processing