StreamGuard: Low-Overhead Resilience for Real-time HPC Data Streams

2026-06-29

AI Summary

This work addresses the challenge of maintaining both timeliness and stability in real-time data streams within scientific workflows, which are highly susceptible to hardware failures, network disruptions, and performance fluctuations in complex environments. The authors propose a lightweight, non-intrusive fault-tolerance mechanism that integrates asynchronous, non-blocking checkpointing with a progress-aware dynamic load redistribution strategy. This approach enables efficient fault recovery and resource rebalancing without interrupting ongoing computations. Under fault-free conditions, the method incurs less than 1% runtime overhead, while in high-failure-rate scenarios, it reduces the impact of faults and performance anomalies by up to sixfold, substantially enhancing the resilience and resource utilization of stream processing systems.

Abstract

Real-time scientific workflows operate on continuous data streams and must produce timely, high-quality results despite executing on complex, failure-prone infrastructure. Hardware faults, network disruptions, and performance anomalies caused by resource contention or system heterogeneity can severely degrade performance and violate real-time constraints. We focus on strengthening the resilience of the producer-consumer streaming pattern, a fundamental building block of scientific streaming workflows. We present two complementary techniques: (i) a dynamic, asynchronous, non-blocking checkpointing mechanism that preserves progress without interrupting computation, and (ii) a progress-aware load redistribution strategy that detects slow workers and proactively rebalances tasks. Together, these mechanisms maintain forward progress and balanced execution even in highly error-prone environments. Experimental results show that our approach reduces the impact of failures and performance anomalies by up to 6x, while introducing less than 1% overhead in failure-free execution.

Problem

• resilience
• real-time
• HPC data streams
• failure recovery
• performance anomalies

Innovation

• non-blocking checkpointing
• progress-aware load balancing
• real-time HPC streams
• fault resilience
• asynchronous recovery