FROSS: Faster-than-Real-Time Online 3D Semantic Scene Graph Generation from RGB-D Images

2025-07-26

AI Summary

Existing 3D semantic scene graph (SSG) generation methods suffer from high computational overhead and lack incremental update capability, hindering real-time deployment in open-world settings. This paper introduces FROSS, the first framework enabling sub-real-time online 3D SSG generation: it employs a lightweight 2D scene graph-to-3D spatial mapping, models object geometry and semantics via 3D Gaussian distributions, and eliminates reliance on dense point clouds; further, it designs an incremental graph update mechanism to support dynamic, structured understanding under streaming RGB-D input. To facilitate rigorous evaluation, we introduce ReplicaSSG—the first benchmark dataset tailored for open-scene 3D SSG generation. Experiments demonstrate that FROSS significantly outperforms state-of-the-art methods on both ReplicaSSG and 3DSSG, achieving real-time inference speed and exhibiting strong potential for deployment in real-world open environments.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The ability to abstract complex 3D environments into simplified and structured representations is crucial across various domains. 3D semantic scene graphs (SSGs) achieve this by representing objects as nodes and their interrelationships as edges, facilitating high-level scene understanding. Existing methods for 3D SSG generation, however, face significant challenges, including high computational demands and non-incremental processing that hinder their suitability for real-time open-world applications. To address this issue, we propose FROSS (Faster-than-Real-Time Online 3D Semantic Scene Graph Generation), an innovative approach for online and faster-than-real-time 3D SSG generation that leverages the direct lifting of 2D scene graphs to 3D space and represents objects as 3D Gaussian distributions. This framework eliminates the dependency on precise and computationally-intensive point cloud processing. Furthermore, we extend the Replica dataset with inter-object relationship annotations, creating the ReplicaSSG dataset for comprehensive evaluation of FROSS. The experimental results from evaluations on ReplicaSSG and 3DSSG datasets show that FROSS can achieve superior performance while operating significantly faster than prior 3D SSG generation methods. Our implementation and dataset are publicly available at https://github.com/Howardkhh/FROSS.

Problem

• High computational demands in 3D SSG generation
• Non-incremental processing hinders real-time applications
• Lack of efficient online 3D scene abstraction

Innovation

• Lifts 2D scene graphs to 3D space
• Represents objects as 3D Gaussian distributions
• Eliminates intensive point cloud processing