Low-Latency Sliding Window Connectivity

2024-10-01

arXiv.org

AI Summary

Existing methods for real-time connectivity queries over streaming graph data under sliding windows suffer from high indexing update latency during dynamic edge insertions and deletions, failing to meet stringent real-time requirements. This paper proposes a lightweight indexing mechanism based on dynamic spanning tree maintenance. It introduces a novel edge-deletion strategy that eliminates the need for expensive replacement-edge search—previously inherent in traditional spanning tree updates—while synergistically integrating sliding-window semantics with incremental tree restructuring to significantly enhance indexing efficiency. Experimental results demonstrate that our approach reduces indexing update latency by up to 458×, improves throughput by 8×, and substantially decreases memory footprint. It thus achieves comprehensive superiority over state-of-the-art methods across latency, throughput, and space overhead.

Technology Category

• Data Mining & Knowledge Management: Data Stream Mining
• Search and Optimization: Distributed Search
• Machine Learning: Graph-based Machine Learning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

Connectivity queries, which check whether vertices belong to the same connected component, are fundamental in graph computations. Sliding window connectivity processes these queries over sliding windows, facilitating real-time streaming graph analytics. However, existing methods struggle with low-latency processing due to the significant overhead of continuously updating index structures as edges are inserted and deleted. We introduce a novel approach that leverages spanning trees to efficiently process queries. The novelty of this method lies in its ability to maintain spanning trees efficiently as window updates occur. Notably, our approach completely eliminates the need for replacement edge searches, a traditional bottleneck in managing spanning trees during edge deletions. We also present several optimizations to maximize the potential of spanning-tree-based indexes. Our comprehensive experimental evaluation shows that index update latency in spanning trees can be reduced by up to $458 imes$ while maintaining query performance, leading to an $8 imes$ improvement in throughput. Our approach also significantly outperforms the state-of-the-art in both query processing and index updates. Additionally, our methods use significantly less memory and demonstrate consistent efficiency across various settings.

Problem

• Dynamic Graphs
• Sliding Window Connectivity Queries
• Real-time Processing

Innovation

• Sliding Window Connectivity Query
• Dynamic Graph Data
• Efficient Index Updating