Event-Grounding Graph: Unified Spatio-Temporal Scene Graph from Robotic Observations

2025-10-21

AI Summary

Existing robotic environment representation methods struggle to model semantic associations between spatial objects and dynamic events (e.g., “blue mug” and “washing”). This paper proposes the Event-Grounded Graph (EGG) framework, the first to jointly integrate semantic scene graphs with event graph neural networks for unified spatiotemporal modeling of static structure and dynamic behavior. EGG constructs temporally consistent, multimodal scene graphs—leveraging semantic segmentation and object detection—to enable service robot–oriented environmental understanding and spatiotemporal query answering. Evaluated on a real-world robotic dataset, EGG significantly improves accuracy in dynamic scene retrieval and natural-language query interpretation. The framework’s source code and dataset are publicly released.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: Scene Analysis & Understanding

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

A fundamental aspect for building intelligent autonomous robots that can assist humans in their daily lives is the construction of rich environmental representations. While advances in semantic scene representations have enriched robotic scene understanding, current approaches lack a connection between spatial features and dynamic events; e.g., connecting the blue mug to the event washing a mug. In this work, we introduce the event-grounding graph (EGG), a framework grounding event interactions to spatial features of a scene. This representation allows robots to perceive, reason, and respond to complex spatio-temporal queries. Experiments using real robotic data demonstrate EGG's capability to retrieve relevant information and respond accurately to human inquiries concerning the environment and events within. Furthermore, the EGG framework's source code and evaluation dataset are released as open-source at: https://github.com/aalto-intelligent-robotics/EGG.

Problem

• Connects spatial features to dynamic events in robotic observations
• Enables robots to perceive and reason about spatio-temporal queries
• Grounds event interactions to spatial scene features for human assistance

Innovation

• Event-Grounding Graph connects events to spatial features
• Framework enables robots to perceive spatio-temporal queries
• Uses real robotic data for accurate environmental responses