Dream-in-Style: Text-to-3D Generation using Stylized Score Distillation

2024-06-05

arXiv.org

AI Summary

This work addresses text-driven stylized 3D generation: reconstructing a style-consistent neural radiance field (NeRF) from a text prompt and a single style reference image, in an end-to-end manner. The method introduces a dual-branch score distillation framework that jointly optimizes NeRF geometry/appearance and style alignment. Its core contributions are: (1) the first integration of style injection into text-to-3D optimization via a novel stylized score distillation loss; and (2) implicit cross-modal style transfer by modulating key-value features in the self-attention layers of a pretrained Stable Diffusion model. Experiments demonstrate statistically significant improvements over state-of-the-art methods (p < 0.01) in both visual quality and style fidelity, while enabling joint synthesis of complex geometry and material properties.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present a method to generate 3D objects in styles. Our method takes a text prompt and a style reference image as input and reconstructs a neural radiance field to synthesize a 3D model with the content aligning with the text prompt and the style following the reference image. To simultaneously generate the 3D object and perform style transfer in one go, we propose a stylized score distillation loss to guide a text-to-3D optimization process to output visually plausible geometry and appearance. Our stylized score distillation is based on a combination of an original pretrained text-to-image model and its modified sibling with the key and value features of self-attention layers manipulated to inject styles from the reference image. Comparisons with state-of-the-art methods demonstrated the strong visual performance of our method, further supported by the quantitative results from our user study.

Problem

• Generate 3D objects from text prompts
• Transfer styles using reference images
• Optimize geometry and appearance via stylized distillation

Innovation

• Text-to-3D generation
• Stylized score distillation
• Neural radiance field reconstruction