Cast: Automated Resilience Testing for Production Cloud Service Systems

2026-02-01

AI Summary

This work addresses the challenge of effectively validating system resilience in microservice architectures, where distributed complexity renders traditional manual testing inadequate. To this end, the authors propose an end-to-end automated resilience testing framework that replays real production traffic and injects application-level faults within a structured three-phase pipeline—initialization, fault injection, and recovery—coupled with a multi-dimensional evaluation mechanism to systematically assess resilience. The framework innovatively incorporates complexity-driven test pruning and prioritization strategies to efficiently cover critical execution paths. Deployed over eight months across four large-scale Huawei Cloud applications handling millions of calls, the approach identified 137 potential resilience vulnerabilities (89 confirmed) and achieved 90% detection coverage on 48 reproducible defects, significantly enhancing both testing fidelity and efficiency.

Technology Category

• Search and Optimization: Distributed Search
• Machine Learning: Scalability of ML Systems
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The distributed nature of microservice architecture introduces significant resilience challenges. Traditional testing methods, limited by extensive manual effort and oversimplified test environments, fail to capture production system complexity. To address these limitations, we present Cast, an automated, end-to-end framework for microservice resilience testing in production. It achieves high test fidelity by replaying production traffic against a comprehensive library of application-level faults to exercise internal error-handling logic. To manage the combinatorial test space, Cast employs a complexity-driven strategy to systematically prune redundant tests and prioritize high-value tests targeting the most critical service execution paths. Cast automates the testing lifecycle through a three-phase pipeline (i.e., startup, fault injection, and recovery) and uses a multi-faceted oracle to automatically verify system resilience against nuanced criteria. Deployed in Huawei Cloud for over eight months, Cast has been adopted by many service teams to proactively address resilience vulnerabilities. Our analysis on four large-scale applications with millions of traces reveals 137 potential vulnerabilities, with 89 confirmed by developers. To further quantify its performance, Cast is evaluated on a benchmark set of 48 reproduced bugs, achieving a high coverage of 90%. The results show that Cast is a practical and effective solution for systematically improving the reliability of industrial microservice systems.

Problem

• microservice resilience
• production testing
• fault injection
• system reliability
• distributed systems

Innovation

• resilience testing
• microservice architecture
• production traffic replay
• fault injection
• automated testing