D3D-GEN: Robot-Aware Domain-Grounded Interactive 3D World Generation for Social Robotics

2026-08-12

AI Summary

Existing simulation environments for social robots struggle to balance realism with simulatability and lack efficient methods for generating semantically plausible, diverse, and fully interactive 3D scenes. This work proposes D3D-GEN, a novel system that achieves, for the first time, fully automated generation of simulation environments driven solely by domain knowledge without relying on predefined 3D model libraries. By integrating domain-specific agents with a retrieval-augmented generation (RAG) pipeline, D3D-GEN automatically constructs a persistent semantic database and dynamically queries it to produce coherent floorplans and object arrangements. The system supports multiple domains—including residential, hospital, and office settings—and generates dozens of structurally varied, semantically consistent, and interactable 3D worlds per domain, seamlessly integrating with mainstream robotic simulators such as Isaac Sim and Gazebo.

Abstract

Training and validation of Embodied AI for social navigation critically depends on realistic simulation environments, yet many current approaches fail to find a balance between realism and simulability. We propose D3D-GEN, a novel world generation system that combines a domain agent with a retrieval-augmented generation (RAG) pipeline grounded in that domain. Our system enables users to rapidly generate domain-grounded, fully interactive 3D worlds by automating both the collection of domain knowledge and the synthesis of realistic floorplans and object placements, without dependence on any fixed 3D model database. Given a domain description prompt, the research agent collects publicly accessible domain-specific data and constructs a persistent domain database. Using this database, our RAG pipeline generates plausible floorplans and object placements by dynamically querying a user-provided semantic database, which can be easily extended or modified. The output is a fully interactive 3D world loadable by the popular simulators Isaac Sim and Gazebo. With our approach, we have built databases for several common domains (indoor residential, hospital, office) and generated dozens of distinct, plausible simulation environments for each domain. We present D3D-GEN with a local web frontend that facilitates rapid, interactive world generation for robot simulation.

Problem

• social robotics
• embodied AI
• simulation environments
• realism
• simulability

Innovation

• domain-grounded generation
• retrieval-augmented generation (RAG)
• interactive 3D world generation
• robot simulation
• social robotics