SceneMotifCoder: Example-driven Visual Program Learning for Generating 3D Object Arrangements

2024-08-05

arXiv.org

AI Summary

Existing text-to-3D approaches struggle to generate multi-object, physically plausible, and semantically aligned 3D scene layouts. This paper proposes an exemplar-driven visual program learning framework—the first to integrate large language models (LLMs) with visual program synthesis for 3D layout modeling. Our method induces compact, editable meta-program representations from few-shot examples, enabling cross-scene generalization and user-controllable editing. By jointly leveraging 3D object retrieval and geometry-aware optimization, it achieves structurally controllable, object-variable, high-fidelity layout generation. Experiments demonstrate that our approach significantly outperforms state-of-the-art text-to-3D and layout generation methods in both text alignment and physical plausibility. Notably, it attains high-fidelity layouts using only a small number of exemplars, establishing a new paradigm for data-efficient, interpretable, and interactive 3D scene synthesis.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Despite advances in text-to-3D generation methods, generation of multi-object arrangements remains challenging. Current methods exhibit failures in generating physically plausible arrangements that respect the provided text description. We present SceneMotifCoder (SMC), an example-driven framework for generating 3D object arrangements through visual program learning. SMC leverages large language models (LLMs) and program synthesis to overcome these challenges by learning visual programs from example arrangements. These programs are generalized into compact, editable meta-programs. When combined with 3D object retrieval and geometry-aware optimization, they can be used to create object arrangements varying in arrangement structure and contained objects. Our experiments show that SMC generates high-quality arrangements using meta-programs learned from few examples. Evaluation results demonstrates that object arrangements generated by SMC better conform to user-specified text descriptions and are more physically plausible when compared with state-of-the-art text-to-3D generation and layout methods.

Problem

• Generating physically plausible multi-object 3D arrangements
• Overcoming failures in text-to-3D arrangement accuracy
• Learning compact visual programs from few examples

Innovation

• Leverages LLMs and program synthesis
• Generalizes examples into editable meta-programs
• Combines retrieval and geometry-aware optimization