Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis

2026-07-06

AI Summary

This work addresses the limitation of existing open-vocabulary 3D scene understanding methods, which often rely on context-free semantic representations and neglect the critical role of inter-object relationships in semantic refinement. To overcome this, the authors propose a relationship-aware 3D scene graph construction approach that requires no manual relation annotations. The method leverages vision-language reasoning to infer object relationships and employs multi-view geometric constraints to eliminate implausible connections. Furthermore, an adaptive gated dual-stream graph attention network is introduced to disentangle and fuse geometric and semantic features effectively. Hierarchical contrastive learning is incorporated to enhance both instance-level consistency and category-level discriminability. Evaluated on multiple benchmarks—including ScanNetV2, ScanNet200, ScanNet++, and Replica—the proposed approach significantly improves open-vocabulary 3D understanding performance and generalization capability.

Abstract

Open-vocabulary 3D scene understanding aims to segment 3D scenes beyond predefined categories by transferring semantic knowledge from vision-language models. Existing methods have advanced this task by lifting language-aligned 2D features into 3D, yet they often rely on context-independent semantic representations, leaving object relationships underexplored for contextual refinement. We propose RelGraphOV, a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding. Our method constructs relational scene graphs from multi-view observations by leveraging vision-language reasoning to infer object relationships and prune geometrically implausible connections, without manual relationship annotations. To aggregate relational context while avoiding feature interference, we introduce an Adaptive Gated Dual-Stream Contextual GAT that separates dense geometric features and semantic CLIP embeddings, performs edge-guided message passing, and adaptively fuses complementary semantics. A hierarchical contrastive objective further promotes instance-level consistency and category-level discrimination. Experiments on ScanNetV2, ScanNet200, ScanNet$++$, and Replica demonstrate strong performance and generalization ability. Project Page: https://cxavireh.github.io/relgraphov-projectpage

Problem

• open-vocabulary 3D scene understanding
• object relationships
• 3D scene graph
• contextual refinement
• vision-language models

Innovation

• relationship-aware
• 3D scene graph
• open-vocabulary 3D understanding
• vision-language reasoning
• contextual GAT