🤖 AI Summary
This work reveals that benchmark characteristics—such as program scale and initial seed coverage—significantly influence fuzzer rankings, challenging the stability assumptions underlying current fuzzing evaluations. To address this, we propose an interpretable evaluation framework grounded in causal attribution and multivariate sensitivity analysis, which quantifies the relative contributions of benchmark attributes versus fuzzer selection to evaluation outcomes. Our framework integrates coverage modeling with cross-fuzzer comparative experiments. Systematic analysis uncovers, for the first time, that AFL’s ranking improves on large-scale programs or with high-coverage seeds, whereas LibFuzzer’s ranking degrades markedly under the same conditions. We further develop a reproducible tool for quantifying evaluation bias, enabling generalizability diagnostics and context-aware fuzzer selection. This advances the credibility and practical utility of fuzzing evaluation methodologies.
📝 Abstract
While the aim of fuzzer evaluation is to establish fuzzer performance in general, an evaluation is always conducted on a specific benchmark. In this paper, we investigate the degree to which the benchmarking result depends on the properties of the benchmark and propose a methodology to quantify the impact of benchmark properties on the benchmarking result in relation to the impact of the choice of fuzzer. We found that the measured performance and ranking of a fuzzer substantially depends on properties of the programs and the seed corpora used during evaluation. For instance, if the benchmark contained larger programs or seed corpora with a higher initial coverage, AFL's ranking would improve while LibFuzzer's ranking would worsen. We describe our methodology as explainable fuzzer evaluation because it explains why the specific evaluation setup yields the observed superiority or ranking of the fuzzers and how it might change for different benchmarks. We envision that our analysis can be used to assess the degree to which evaluation results are overfitted to the benchmark and to identify the specific conditions under which different fuzzers performs better than others.
