PAT3D: Physics-Augmented Text-to-3D Scene Generation

2025-11-26

AI Summary

This work addresses physical implausibility, inter-object penetration, and semantic misalignment in text-to-3D scene generation. We propose the first end-to-end framework integrating vision-language models (VLMs) with differentiable rigid-body simulation. Methodologically: (1) a VLM parses input text to construct a hierarchical semantic scene graph; (2) differentiable rigid-body dynamics simulate physical interactions, enabling joint optimization of object poses and support relationships; (3) static equilibrium and non-penetration constraints jointly optimize geometric layout and physical state. Our key contribution is the first physically plausible, simulation-ready text-driven 3D scene generation method—requiring no post-processing for downstream robotic manipulation or scene editing. Experiments demonstrate significant improvements over prior art in physical plausibility, semantic consistency, and visual fidelity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

We introduce PAT3D, the first physics-augmented text-to-3D scene generation framework that integrates vision-language models with physics-based simulation to produce physically plausible, simulation-ready, and intersection-free 3D scenes. Given a text prompt, PAT3D generates 3D objects, infers their spatial relations, and organizes them into a hierarchical scene tree, which is then converted into initial conditions for simulation. A differentiable rigid-body simulator ensures realistic object interactions under gravity, driving the scene toward static equilibrium without interpenetrations. To further enhance scene quality, we introduce a simulation-in-the-loop optimization procedure that guarantees physical stability and non-intersection, while improving semantic consistency with the input prompt. Experiments demonstrate that PAT3D substantially outperforms prior approaches in physical plausibility, semantic consistency, and visual quality. Beyond high-quality generation, PAT3D uniquely enables simulation-ready 3D scenes for downstream tasks such as scene editing and robotic manipulation. Code and data will be released upon acceptance.

Problem

• Generates physically plausible 3D scenes from text prompts.
• Ensures object interactions are realistic and free of intersections.
• Produces simulation-ready scenes for downstream applications like robotics.

Innovation

• Integrates vision-language models with physics-based simulation
• Uses differentiable rigid-body simulator for realistic interactions
• Employs simulation-in-the-loop optimization for stability and consistency