Co-generation of Layout and Shape from Text via Autoregressive 3D Diffusion

2026-04-17

AI Summary

Existing text-to-scene generation methods struggle to simultaneously achieve layout coherence and geometric consistency of objects under complex semantic conditions. This work proposes a sequential generation paradigm that, for the first time, unifies autoregressive sequence modeling with 3D diffusion mechanisms to jointly synthesize scene layouts and detailed object geometries in a staged, collaborative manner across scene and object spaces. Leveraging multimodal token sequences as conditioning inputs, we introduce 3D-ARD+, a 3D autoregressive diffusion model trained on 230K indoor scene–text pairs. Experimental results demonstrate that our 7B-parameter model accurately generates and positions objects exhibiting intricate spatial relationships and rich semantics, significantly improving both text-to-scene alignment and geometric fidelity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent text-to-scene generation approaches largely reduced the manual efforts required to create 3D scenes. However, their focus is either to generate a scene layout or to generate objects, and few generate both. The generated scene layout is often simple even with LLM's help. Moreover, the generated scene is often inconsistent with the text input that contains non-trivial descriptions of the shape, appearance, and spatial arrangement of the objects. We present a new paradigm of sequential text-to-scene generation and propose a novel generative model for interactive scene creation. At the core is a 3D Autoregressive Diffusion model 3D-ARD+, which unifies the autoregressive generation over a multimodal token sequence and diffusion generation of next-object 3D latents. To generate the next object, the model uses one autoregressive step to generate the coarse-grained 3D latents in the scene space, conditioned on both the current seen text instructions and already synthesized 3D scene. It then uses a second step to generate the 3D latents in the smaller object space, which can be decoded into fine-grained object geometry and appearance. We curate a large dataset of 230K indoor scenes with paired text instructions for training. We evaluate 7B 3D-ARD+, on challenging scenes, and showcase the model can generate and place objects following non-trivial spatial layout and semantics prescribed by the text instructions.

Problem

• text-to-scene generation
• scene layout
• 3D object generation
• spatial arrangement
• text-conditioned 3D generation

Innovation

• autoregressive diffusion
• text-to-3D scene generation
• co-generation of layout and shape
• multimodal token sequence
• 3D latent generation