OpenScan: A Benchmark for Generalized Open-Vocabulary 3D Scene Understanding

2024-08-20

arXiv.org

AI Summary

Existing open-vocabulary 3D scene understanding (OV-3D) methods focus solely on object category recognition, failing to evaluate models’ comprehension of fine-grained semantic attributes. Method: We propose generalized open-vocabulary 3D scene understanding (GOV-3D), a novel task that extends open-vocabulary reasoning to eight linguistic dimensions—function, material, color, shape, size, orientation, spatial relation, and state—encompassing abstract, object-specific, and fine-grained attribute queries. To support this, we introduce OpenScan, the first 3D benchmark for attribute-level language understanding, built upon real-world ScanNet and Structured3D scans and annotated with a cross-dimensional attribute taxonomy. Contribution/Results: Extensive experiments reveal that state-of-the-art OV-3D models achieve only <32% average accuracy on GOV-3D, exposing critical deficiencies in semantic disentanglement and attribute-aware representation learning—thereby establishing a new evaluation standard and identifying key research directions for future work.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Open-vocabulary 3D scene understanding (OV-3D) aims to localize and classify novel objects beyond the closed set of object classes. However, existing approaches and benchmarks primarily focus on the open vocabulary problem within the context of object classes, which is insufficient in providing a holistic evaluation to what extent a model understands the 3D scene. In this paper, we introduce a more challenging task called Generalized Open-Vocabulary 3D Scene Understanding (GOV-3D) to explore the open vocabulary problem beyond object classes. It encompasses an open and diverse set of generalized knowledge, expressed as linguistic queries of fine-grained and object-specific attributes. To this end, we contribute a new benchmark named extit{OpenScan}, which consists of 3D object attributes across eight representative linguistic aspects, including affordance, property, and material. We further evaluate state-of-the-art OV-3D methods on our OpenScan benchmark and discover that these methods struggle to comprehend the abstract vocabularies of the GOV-3D task, a challenge that cannot be addressed simply by scaling up object classes during training. We highlight the limitations of existing methodologies and explore promising directions to overcome the identified shortcomings.

Problem

• Extends 3D scene understanding beyond predefined object classes.
• Introduces generalized open-vocabulary 3D scene understanding (GOV-3D).
• Evaluates models on fine-grained, object-specific attributes via OpenScan.

Innovation

• Introduces Generalized Open-Vocabulary 3D Scene Understanding
• Develops OpenScan benchmark with diverse 3D attributes
• Evaluates state-of-the-art methods on abstract vocabularies