Nova: Scalable Streaming Join Placement and Parallelization in Resource-Constrained Geo-Distributed Environments

2026-03-16

AI Summary

This work addresses the challenges of efficiently deploying streaming join operations in resource-constrained wide-area edge environments, where high latency, node overloading, and dynamic network changes are prevalent. The authors propose the first scalable joint optimization framework tailored for edge stream processing, transforming the NP-hard operator placement and replication problem into a convex optimization formulation. By integrating Euclidean space embedding with sub-join partitioning, the approach enables low-latency, resource-aware deployment. It further supports localized re-optimization to rapidly adapt to topological changes, with re-optimization time nearly independent of system scale. Experimental results demonstrate up to a 39× reduction in latency and a 4.5× improvement in throughput compared to state-of-the-art methods, while effectively preventing node overload.

Technology Category

• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Real-time data processing in large geo-distributed applications, like the Internet of Things (IoT), increasingly shifts computation from the cloud to the network edge to reduce latency and mitigate network congestion. In this setting, minimizing latency while avoiding node overload requires jointly optimizing operator replication and placement of operator instances, a challenge known as the Operator Placement and Replication (OPR) problem. OPR is NP-hard and particularly difficult to solve in large-scale, heterogeneous, and dynamic geo-distributed networks, where solutions must be scalable, resource-aware, and adaptive to changes like node failures. Existing work on OPR has primarily focused on single-stream operators, such as filters and aggregations. However, many latency-sensitive applications, like environmental monitoring and anomaly detection, require efficient regional stream joins near data sources. This paper introduces Nova, an optimization approach designed to address OPR for join operators that are computable on resource-constrained edge devices. Nova relaxes the NP-hard OPR into a convex optimization problem by embedding cost metrics into a Euclidean space and partitioning joins into smaller sub-joins. This new formulation enables linear scalability and efficient adaptation to topological changes through partial re-optimizations. We evaluate Nova through simulations on real-world topologies and on a local testbed, demonstrating up to 39x latency reduction and 4.5x increase in throughput compared to existing edge-centered solutions, while also preventing node overload and maintaining near-constant re-optimization times regardless of topology size.

Problem

• Operator Placement and Replication
• streaming join
• geo-distributed
• edge computing
• resource-constrained

Innovation

• Operator Placement and Replication
• Stream Join
• Convex Optimization
• Edge Computing
• Geo-Distributed Systems