MUSE: Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfaction

2026-06-12

AI Summary

Existing text-driven 3D scene generation methods lack the ability to track user intent over time and perform localized edits, often requiring full regeneration or manual intervention. This work proposes the first incremental 3D scene authoring framework that integrates a memory mechanism with multi-agent collaboration, modeling creation as an iterative process of progressively satisfying user requirements. Specifically, an Architect agent parses natural language instructions into structured goals, a Sculptor agent executes localized modifications, and an Inspector agent validates outcomes and updates three types of memory—working, scene, and skill memory. Evaluated on AuthorBench, the method achieves an 80.7% success rate on full construction tasks, retains 99.9% of existing content during editing, and introduces only 0.6% unintended changes, significantly outperforming current baselines.

Abstract

Text-driven 3D scene generation is a promising technique for digital content creation, embodied AI simulation, and interactive design, yet practical workflows often require refining, extending, or correcting existing scenes while preserving non-target content. Existing methods can produce realistic and structurally plausible scenes, but they generally lack editability with requirement-level state tracking, so part-level failures often lead to full-scene regeneration or manual intervention. To tackle this challenge, we formulate controllable 3D scene authoring as incremental requirement satisfaction, unifying construction and editing. In this paper, we present MUSE, a memory-grounded multi-agent framework in which an Architect compiles instructions into structured requirements, a Sculptor executes local scene operations, and an Inspector verifies each step while updating Working, Scene, and Skill Memory. To evaluate requirement-level controllability and preservation-aware editing, we introduce AuthorBench, offering 145 constrained construction cases and a 1,584-case preservation-aware editing pool paired with external structured checks. On full construction cases, MUSE improves All-Goal success from 37.9 to 80.7 and surface-constraint fulfillment from 35.0 to 92.6 over the strongest baseline. On a stratified 240-case editing test split, MUSE achieves 49.6 All-Goal success, 99.9 preservation rate, and only 0.6 unintended change rate. Beyond automated metrics, human evaluations on compared local-editing baselines support stronger alignment with user intent, and downstream navigation-proxy tests indicate stronger spatial stability. Combined with ablations validating our memory designs, these results establish MUSE as an effective framework for controllable 3D scene authoring.

Problem

• 3D scene editing
• requirement-level controllability
• preservation-aware editing
• text-driven generation
• scene authoring

Innovation

• incremental requirement satisfaction
• memory-grounded multi-agent
• preservation-aware editing
• controllable 3D scene authoring
• structured requirement compilation