AgileDART: An Agile and Scalable Edge Stream Processing Engine

2024-07-20

IEEE Transactions on Mobile Computing

AI Summary

To address the challenges of low concurrency, high latency, and poor adaptability in conventional stream processing systems—stemming from constrained computational resources, unstable network conditions, and massive, heterogeneous, dynamically evolving sensor data on edge devices—this paper proposes a lightweight, adaptive stream query execution framework for edge computing. Our approach introduces two key innovations: (1) a dynamic dataflow abstraction coupled with a DHT-based peer-to-peer operator orchestration mechanism to enable decentralized, elastic scheduling; and (2) an online path re-planning model integrating multi-armed bandit (MAB) learning to optimize data routing under network instability. Experimental evaluation demonstrates that, compared to Storm and EdgeWise, our framework significantly reduces end-to-end query latency while enhancing scalability and environmental adaptability for large-scale edge streaming applications.

Technology Category

• Data Mining & Knowledge Management: Data Stream Mining
• Machine Learning: Learning on the Edge & Model Compression
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Edge applications generate a large influx of sensor data on massive scales, and these massive data streams must be processed shortly to derive actionable intelligence. However, traditional data processing systems are not well-suited for these edge applications as they often do not scale well with a large number of concurrent stream queries, do not support low-latency processing under limited edge computing resources, and do not adapt to the level of heterogeneity and dynamicity commonly present in edge computing environments. As such, we present AgileDart, an agile and scalable edge stream processing engine that enables fast stream processing of many concurrently running low-latency edge applications' queries at scale in dynamic, heterogeneous edge environments. The novelty of our work lies in a dynamic dataflow abstraction that leverages distributed hash table-based peer-to-peer overlay networks to autonomously place, chain, and scale stream operators to reduce query latencies, adapt to workload variations, and recover from failures and a bandit-based path planning model that re-plans the data shuffling paths to adapt to unreliable and heterogeneous edge networks. We show that AgileDart outperforms Storm and EdgeWise on query latency and significantly improves scalability and adaptability when processing many real-world edge stream applications' queries.

Problem

• Edge Computing
• High Concurrency
• Resource Limitations

Innovation

• AgileDART
• Adaptive Edge Computing
• Dynamic Workload Management