Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptions

2025-07-19

AI Summary

Current LLM-based 3D scene understanding methods struggle to accurately model complex spatial and semantic relationships among objects—particularly when visual embeddings alone are insufficient to capture functional roles and interactive affordances. To address this, we propose a language-guided relational modeling framework that leverages object-level textual descriptions. Our approach introduces a two-tier fusion mechanism: (i) embedding-level fusion of 2D/3D visual features with text embeddings, and (ii) prompt-level injection of structured relational priors to enable explicit natural-language grounding of 3D object relations. Crucially, the framework requires no task-specific heads or additional supervision, supporting unified cross-task inference. We evaluate on five major benchmarks—ScanRefer, Multi3DRefer, ScanQA, SQA3D, and Scan2Cap—and consistently outperform strong baselines, demonstrating both the effectiveness and generalizability of language-guided relational representation in 3D scene understanding.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Understanding 3D scenes goes beyond simply recognizing objects; it requires reasoning about the spatial and semantic relationships between them. Current 3D scene-language models often struggle with this relational understanding, particularly when visual embeddings alone do not adequately convey the roles and interactions of objects. In this paper, we introduce Descrip3D, a novel and powerful framework that explicitly encodes the relationships between objects using natural language. Unlike previous methods that rely only on 2D and 3D embeddings, Descrip3D enhances each object with a textual description that captures both its intrinsic attributes and contextual relationships. These relational cues are incorporated into the model through a dual-level integration: embedding fusion and prompt-level injection. This allows for unified reasoning across various tasks such as grounding, captioning, and question answering, all without the need for task-specific heads or additional supervision. When evaluated on five benchmark datasets, including ScanRefer, Multi3DRefer, ScanQA, SQA3D, and Scan2Cap, Descrip3D consistently outperforms strong baseline models, demonstrating the effectiveness of language-guided relational representation for understanding complex indoor scenes.

Problem

• Enhancing 3D scene understanding with object-level text descriptions
• Improving relational reasoning between objects in 3D scenes
• Unifying multiple 3D tasks without task-specific supervision

Innovation

• Uses object-level text descriptions for 3D scenes
• Integrates relational cues via dual-level fusion
• Enables unified reasoning without task-specific heads