Detecting and Mitigating Flakiness in REST API Fuzzing

2026-03-30

AI Summary

This study addresses the critical yet underexplored issue of flakiness in REST API testing, which severely undermines the reliability of automated test suites. Through an empirical analysis of nearly 3,000 failing test cases across 36 real-world APIs, the work systematically identifies and categorizes the root causes of flakiness in REST API fuzzing. Building on these insights, the authors propose FlakyCatch, a general-purpose approach capable of effectively detecting and mitigating flakiness in tests generated by both white-box and black-box fuzzers, including EvoMaster. Experimental evaluation demonstrates that FlakyCatch substantially enhances test stability and reliability, establishing a new paradigm for robust REST API testing.

Abstract

Test flakiness is a common problem in industry, which hinders the reliability of automated build and testing workflows. Most existing research on test flakiness has primarily focused on unit and small-scale integration tests. In contrast, flakiness in system-level testing such as REST APIs are comparatively under-explored. A large body of literature has been dedicated to the topic of fuzzing REST APIs, whereas relatively little attention has been paid to detecting and possibly mitigating negative effects of flakiness in this context. To fill this major gap, in this paper, we study the flakiness of tests generated by one of the popularly applied REST API fuzzer in the literature, namely EvoMaster, conduct empirical studies with a corpus of 36 REST APIs to understand flakiness of REST APIs. Based on the results of the empirical studies, we categorize and analyze flakiness sources by inspecting near 3000 failing tests. Based on the understanding, we propose FlakyCatch to detect and mitigate flakiness in REST APIs and empirically evaluate its performance. Results show that FlakyCatch is effective in detecting and handling flakiness in tests generated by white-box and black-box fuzzers.

Problem

• test flakiness
• REST API
• fuzzing
• system-level testing
• automated testing

Innovation

• test flakiness
• REST API fuzzing
• FlakyCatch
• empirical study
• automated testing