Generalised Rate Control Approach For Stream Processing Applications

2025-06-13

AI Summary

Distributed stream processing systems often suffer from overload during real-time data surges, leading to state instability and resource underutilization. To address this, we propose a graph neural network (GNN)-enhanced deep reinforcement learning (DRL) framework for coordinated flow control. This work is the first to incorporate GNNs into flow control decision-making: it constructs a dynamic graph representation from system topology and runtime monitoring metrics, enabling joint, low-latency, state-agnostic regulation of multi-source data emission rates. The approach is cross-application and multi-scenario adaptive, requiring neither explicit state modeling nor historical feedback. Evaluated on three representative stream processing applications, our method achieves a 13.5% throughput improvement and a 30% reduction in end-to-end latency—significantly outperforming conventional heuristic and vanilla DRL baselines.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Natural Language Processing: Learning & Optimization for NLP
• Data Mining & Knowledge Management: Data Stream Mining

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Distributed stream processing systems are widely deployed to process real-time data generated by various devices, such as sensors and software systems. A key challenge in the system is overloading, which leads to an unstable system status and consumes additional system resources. In this paper, we use a graph neural network-based deep reinforcement learning to collaboratively control the data emission rate at which the data is generated in the stream source to proactively avoid overloading scenarios. Instead of using a traditional multi-layer perceptron-styled network to control the rate, the graph neural network is used to process system metrics collected from the stream processing engine. Consequently, the learning agent (i) avoids storing past states where previous actions may affect the current state, (ii) is without waiting a long interval until the current action has been fully effective and reflected in the system's specific metrics, and more importantly, (iii) is able to adapt multiple stream applications in multiple scenarios. We deploy the rate control approach on three applications, and the experimental results demonstrate that the throughput and end-to-end latency are improved by up to 13.5% and 30%, respectively.

Problem

• Controls data emission rate to prevent system overloading
• Uses graph neural network for adaptive rate control
• Improves throughput and latency in stream processing

Innovation

• Graph neural network for rate control
• Deep reinforcement learning avoids overloading
• Adapts to multiple stream applications