DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priors

2026-07-01

AI Summary

Existing approaches struggle to construct spatiotemporally consistent 3D semantic scene graphs due to unstable 3D object representations and frame-by-frame inference. This work proposes a novel framework that first models objects as probabilistic 3D nodes by estimating instance-level geometric 3D Gaussian distributions via depth-guided filtering. It then leverages contextual priors from the V-JEPA 2 world model to aggregate relational evidence across space and time, yielding robust 3D semantic scene graphs. By uniquely integrating depth-aware probabilistic node representations with world-model-derived relational priors, the method substantially mitigates relation sparsity and temporal inconsistency. It achieves state-of-the-art performance on 3DSSG and ReplicaSSG, improving triplet and predicate recall by 77.4% and 23.2%, respectively, while producing structurally coherent outputs suitable for robotic manipulation and augmented reality applications.

Abstract

We present DeWorldSG, a novel framework that generates spatio-temporally robust 3D Semantic Scene Graphs from RGB-D sequences. Existing methods often struggle to construct reliable 3D scene graphs due to unstable 3D object representations and missing relations caused by frame-wise inference. DeWorldSG addresses these issues by estimating instance-level geometric 3D Gaussian distributions through depth-guided filtering and representing each object as a probabilistic 3D node rather than a single projected point. To mitigate relational sparsity from frame-wise inference, our framework further aggregates spatiotemporal evidence across object pairs and refines relations using contextual priors derived from a world model (V-JEPA 2). Experiments on the 3DSSG and ReplicaSSG datasets demonstrate state-of-the-art (SoTA) performance in both object and predicate prediction, while producing temporally consistent scene structures. In particular, our method improves triplet recall by 77.4% and predicate recall by 23.2% over prior SoTA approaches, making it suitable for robotic manipulation and AR applications. Our code and models are open-sourced.

Problem

• 3D Semantic Scene Graph
• RGB-D sequences
• spatio-temporal consistency
• relational sparsity
• 3D object representation

Innovation

• 3D Semantic Scene Graph
• Depth-Aware Representation
• Probabilistic 3D Node
• Spatiotemporal Aggregation
• World-Model Priors