HouseCrafter: Lifting Floorplans to 3D Scenes with 2D Diffusion Model

2024-06-28

arXiv.org

AI Summary

This work addresses the problem of generating high-fidelity, structurally coherent large-scale 3D indoor scenes (e.g., whole houses) from 2D floor plans. Methodologically, it pioneers the adaptation of web-scale pre-trained 2D diffusion models to 3D scene generation, introducing a floor-plan-conditioned encoding scheme and an attention-driven consistency modeling mechanism. It autoregressively synthesizes multi-view-consistent RGB-D images while incorporating global floor-plan guidance to ensure spatial coherence. The key contribution lies in leveraging 2D diffusion priors for joint cross-view geometric and appearance modeling—enabling high-fidelity texture synthesis and plausible structural layout without any 3D supervision. Experiments on the 3D-Front dataset demonstrate that the generated house-level scenes significantly outperform existing methods in structural plausibility, visual realism, and multi-view consistency. Ablation studies validate the efficacy of each proposed component.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

We introduce HouseCrafter, a novel approach that can lift a floorplan into a complete large 3D indoor scene (e.g., a house). Our key insight is to adapt a 2D diffusion model, which is trained on web-scale images, to generate consistent multi-view color (RGB) and depth (D) images across different locations of the scene. Specifically, the RGB-D images are generated autoregressively in a batch-wise manner along sampled locations based on the floorplan, where previously generated images are used as condition to the diffusion model to produce images at nearby locations. The global floorplan and attention design in the diffusion model ensures the consistency of the generated images, from which a 3D scene can be reconstructed. Through extensive evaluation on the 3D-Front dataset, we demonstrate that HouseCraft can generate high-quality house-scale 3D scenes. Ablation studies also validate the effectiveness of different design choices. We will release our code and model weights. Project page: https://neu-vi.github.io/houseCrafter/

Problem

• Lifting 2D floorplans to 3D indoor scenes
• Generating consistent multi-view RGB-D images
• Reconstructing 3D scenes from diffusion-generated imagery

Innovation

• Adapts 2D diffusion model for 3D scene generation
• Uses autoregressive RGB-D image generation with conditioning
• Ensures consistency through floorplan-guided attention design