StreamShield: A Production-Proven Resiliency Solution for Apache Flink at ByteDance

2026-02-03

AI Summary

This work addresses the challenges of slow failure recovery, poor stability, and high operational costs that Apache Flink faces in large-scale production environments, which hinder its ability to meet stringent service-level objectives (SLOs). To overcome these limitations, the authors propose the first systematic approach that integrates engine-level and cluster-level elasticity. The solution innovatively combines runtime optimizations, fine-grained fault tolerance, a hybrid replication strategy, and high-availability mechanisms leveraging external systems, complemented by a highly reliable automated testing and deployment pipeline. Evaluated on ByteDance’s ultra-large-scale Flink clusters, the proposed framework significantly enhances system elasticity, stability, and recovery efficiency, thereby effectively ensuring compliance with demanding SLOs.

Technology Category

• Machine Learning: Scalability of ML Systems
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Search and Optimization: Distributed Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Distributed Stream Processing Systems (DSPSs) form the backbone of real-time processing and analytics at ByteDance, where Apache Flink powers one of the largest production clusters worldwide. Ensuring resiliency, the ability to withstand and rapidly recover from failures, together with operational stability, which provides consistent and predictable performance under normal conditions, is essential for meeting strict Service Level Objectives (SLOs). However, achieving resiliency and stability in large-scale production environments remains challenging due to the cluster scale, business diversity, and significant operational overhead. In this work, we present StreamShield, a production-proven resiliency solution deployed in ByteDance's Flink clusters. Designed along complementary perspectives of the engine and cluster, StreamShield introduces key techniques to enhance resiliency, covering runtime optimization, fine-grained fault-tolerance, hybrid replication strategy, and high availability under external systems. Furthermore, StreamShield proposes a robust testing and deployment pipeline that ensures reliability and robustness in production releases. Extensive evaluations on a production cluster demonstrate the efficiency and effectiveness of techniques proposed by StreamShield.

Problem

• resiliency
• operational stability
• distributed stream processing
• fault tolerance
• Service Level Objectives

Innovation

• resiliency
• fine-grained fault-tolerance
• hybrid replication
• runtime optimization
• high availability