WFC/WFD: Web Fuzzing Commons, Dataset and Guidelines to Support Experimentation in REST API Fuzzing

2025-09-01

AI Summary

To address three critical challenges in REST API fuzz testing—(1) inconsistent authentication handling, (2) lack of standardized fault classification, and (3) non-reproducible experimental benchmarks—this paper introduces Web Fuzzing Commons (WFC), a foundational framework. WFC systematically defines a taxonomy of API authentication mechanisms and establishes a standardized fault catalog for REST APIs. It proposes a declarative authentication configuration specification and a normalized API schema. Furthermore, it releases an open-source benchmark comprising 36 deployable APIs with fully containerized, reproducible execution environments. WFC provides a lightweight tool library and multi-fuzzer integration interfaces, enabling fair, cross-tool evaluation—already supporting EvoMaster and other mainstream fuzzers. Empirical evaluation uncovers prevalent assessment biases in prior work and yields actionable guidelines for improving reproducibility. Overall, WFC significantly advances the scientific rigor, comparability, and reproducibility of API fuzz testing research.

Technology Category

• Natural Language Processing: Safety and Robustness
• Application Domains: Mobility, Driving & Flight
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Responsible Web: Consent frameworks and practices on the web
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Fuzzing REST APIs is an important research problem, with practical applications and impact in industry. As such, a lot of research work has been carried out on this topic in the last few years. However, there are three major issues that hinder further progress: how to deal with API authentication; how to catalog and compare different fault types found by different fuzzers; and what to use as case study to facilitate fair comparisons among fuzzers. To address these important challenges, we present Web Fuzzing Commons (WFC) and Web Fuzzing Dataset (WFD). WFC is a set of open-source libraries and schema definitions to declaratively specify authentication info and catalog different types of faults that fuzzers can automatically detect. WFD is a collection of 36 open-source APIs with all necessary scaffolding to easily run experiments with fuzzers, supported by WFC. To show the usefulness of WFC/WFD, a set of experiments is carried out with EvoMaster, a state-of-the-art fuzzer for Web APIs. However, any fuzzer can benefit from WFC and WFD. We compare EvoMaster with other state-of-the-art tools such as ARAT-RL, EmRest, LLamaRestTest, RESTler, and Schemathesis. We discuss common pitfalls in tool comparisons, as well as providing guidelines with support of WFC/WFD to avoid them.

Problem

• Addressing API authentication challenges in REST fuzzing
• Standardizing fault cataloging for fuzzer comparison
• Providing standardized case studies for fair fuzzer evaluation

Innovation

• Open-source libraries for API authentication specification
• Schema definitions to catalog fuzzer-detected fault types
• Dataset of 36 open-source APIs for experimental benchmarking