IL3D: A Large-Scale Indoor Layout Dataset for LLM-Driven 3D Scene Generation

2025-10-13

AI Summary

To address the scarcity of high-quality, diverse training data for LLM-driven 3D indoor scene generation, this work introduces the first large-scale multimodal dataset tailored to this task—comprising 27,816 photorealistic indoor layouts and 29,215 high-fidelity 3D object assets. Each instance is annotated with fine-grained natural language descriptions and six complementary modalities: point clouds, 3D bounding boxes, multi-view RGB/depth/normal images, and semantic masks. We further provide a unified export interface and an authoritative benchmark for evaluation. Leveraging this dataset, we perform multimodal supervised fine-tuning (SFT) of LLMs, achieving significant improvements over state-of-the-art methods in cross-modal understanding and layout generation, with markedly enhanced generalization. This work establishes a foundational data resource and a scalable technical paradigm for 3D scene generation and embodied intelligence.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

In this study, we present IL3D, a large-scale dataset meticulously designed for large language model (LLM)-driven 3D scene generation, addressing the pressing demand for diverse, high-quality training data in indoor layout design. Comprising 27,816 indoor layouts across 18 prevalent room types and a library of 29,215 high-fidelity 3D object assets, IL3D is enriched with instance-level natural language annotations to support robust multimodal learning for vision-language tasks. We establish rigorous benchmarks to evaluate LLM-driven scene generation. Experimental results show that supervised fine-tuning (SFT) of LLMs on IL3D significantly improves generalization and surpasses the performance of SFT on other datasets. IL3D offers flexible multimodal data export capabilities, including point clouds, 3D bounding boxes, multiview images, depth maps, normal maps, and semantic masks, enabling seamless adaptation to various visual tasks. As a versatile and robust resource, IL3D significantly advances research in 3D scene generation and embodied intelligence, by providing high-fidelity scene data to support environment perception tasks of embodied agents.

Problem

• Addressing the need for diverse indoor layout training data
• Establishing benchmarks for LLM-driven 3D scene generation
• Supporting multimodal learning with rich scene annotations

Innovation

• Large-scale indoor layout dataset for LLM-driven generation
• Enriched with instance-level natural language annotations
• Provides multimodal data exports for various visual tasks