SPARK: Graph-Based Online Semantic Integration System for Robot Task Planning

2025-06-25

AI Summary

Existing robotic systems struggle to efficiently and scalably update semantic information online in dynamic environments, limiting the real-time responsiveness and adaptability of task planning. To address this, we propose SPARK—a novel semantic integration framework that enables online construction and incremental updating of scene graphs. SPARK jointly leverages SLAM-derived geometric constraints and environment embeddings to explicitly model spatial-semantic relationships in a graph-structured representation. Furthermore, it introduces a spatially aware knowledge reasoning mechanism, enabling, for the first time, online recognition and reactive execution based on non-canonical interaction cues (e.g., gestures). Experimental results demonstrate that SPARK significantly improves task execution success rates and generalization capability in complex, dynamic scenarios—particularly excelling in tasks triggered by unconventional perceptual cues.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

The ability to update information acquired through various means online during task execution is crucial for a general-purpose service robot. This information includes geometric and semantic data. While SLAM handles geometric updates on 2D maps or 3D point clouds, online updates of semantic information remain unexplored. We attribute the challenge to the online scene graph representation, for its utility and scalability. Building on previous works regarding offline scene graph representations, we study online graph representations of semantic information in this work. We introduce SPARK: Spatial Perception and Robot Knowledge Integration. This framework extracts semantic information from environment-embedded cues and updates the scene graph accordingly, which is then used for subsequent task planning. We demonstrate that graph representations of spatial relationships enhance the robot system's ability to perform tasks in dynamic environments and adapt to unconventional spatial cues, like gestures.

Problem

• Online updates of semantic information for robots
• Scalable online scene graph representation challenges
• Integrating spatial cues for dynamic task planning

Innovation

• Online semantic scene graph representation
• Dynamic graph updates from environmental cues
• Graph-based spatial relationship task planning