LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models

2024-12-03

arXiv.org

AI Summary

Large language models (LLMs) struggle to generate semantically coherent and physically plausible 3D layouts from natural language instructions in densely constrained physical environments. To address this, we propose a vision-language model (VLM)-driven differentiable 3D layout representation framework that jointly models semantic alignment and physical feasibility in an end-to-end manner—without handcrafted geometric or physical constraints. Our method integrates the VLM’s semantic understanding with differentiable optimization via a dual-path collaborative representation generation and self-consistent spatial decoding mechanism. Furthermore, we fine-tune the VLM on real-world scene data specifically for layout representation, substantially enhancing its spatial reasoning capability. Experiments demonstrate that our approach outperforms both pure LLM-based baselines and conventional constraint-solving methods in both physical plausibility and instruction adherence.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Spatial reasoning is a fundamental aspect of human cognition, enabling intuitive understanding and manipulation of objects in three-dimensional space. However, Large Language Models (LLMs) struggle with simple tasks such as arranging 3D assets in space according to open-ended language instructions, particularly in dense and physically constrained environments. We introduce LayoutVLM, a framework and scene layout representation that exploits the semantic knowledge of Vision-Language Models (VLMs) and supports differentiable optimization to ensure physical plausibility. LayoutVLM employs VLMs to generate two mutually reinforcing representations from visually marked images, and a self-consistent decoding process to improve VLMs spatial planning. Our experiments show that LayoutVLM addresses the limitations of existing LLM and constraint-based approaches, producing physically plausible 3D layouts better aligned with the semantic intent of input language instructions. We also demonstrate that fine-tuning VLMs with the proposed scene layout representation extracted from existing scene datasets can improve their reasoning performance.

Problem

• Enhances 3D spatial reasoning with Vision-Language Models.
• Optimizes 3D layouts for physical plausibility and semantic alignment.
• Improves VLMs' spatial planning through differentiable optimization.

Innovation

• Differentiable optimization for 3D layout
• Vision-Language Models enhance spatial reasoning
• Self-consistent decoding improves spatial planning