Fault Injection in OpenAPI Specifications for Evaluating Black-Box Testing Effectiveness

2026-07-13

AI Summary

This study addresses a critical gap in black-box testing research, which commonly assumes the correctness of OpenAPI specifications while overlooking how their defects impact testing efficacy. The authors propose the first taxonomy of OpenAPI faults, derived from literature, categorizing them into six types, and systematically inject these faults across five severity levels. Using EvoMaster, RESTler, and Schemathesis, they evaluate the effects on two microservice benchmarks through multidimensional metrics—including code and specification coverage, request/response quality, and behavioral diversity. Their findings reveal heterogeneous degradation patterns: method semantics–related faults cause comprehensive performance deterioration, whereas response code modifications have negligible impact. Notably, relaxing schema constraints substantially degrades request/response quality without affecting coverage metrics, demonstrating that reliance solely on coverage can obscure critical quality issues.

Abstract

OpenAPI specifications are the primary input for black-box testing tools in microservice systems (MSS), yet prior work shows these specifications are often incomplete, inconsistent, or incorrect. Despite this, most studies on OpenAPI-based black-box testing assume correct specifications and evaluate tool performance. We address this gap by introducing a literature-grounded taxonomy of six OpenAPI specification fault classes. We inject faults at five severity levels, and evaluate the resulting mutated specifications on two microservice benchmarks, TrainTicket and SocialNetwork, using three testing tools: EvoMaster, RESTler, and Schemathesis. We measure the impact of these faults using code coverage, specification coverage, request/response quality, and behavioral diversity. Our results show that specification faults cause strong and heterogeneous degradation patterns across testing tools and systems. Faults in method semantics cause broad degradation across all metrics, while others, such as modifications to response codes, remain weak. Relaxations of schema constraints cause hidden degradation, with no impact on code and specification coverage but a large impact on request/response quality. These findings demonstrate that specification quality directly shapes black-box API testing effectiveness. Also, code and specification coverage-only evaluations can understate the impact of specification faults on black-box testing in MSS and should be complemented by request/response quality and behavioral diversity.

Problem

• OpenAPI specifications
• black-box testing
• fault injection
• microservice systems
• specification quality

Innovation

• Fault Injection
• OpenAPI Specification
• Black-Box Testing
• Microservice Systems
• Testing Effectiveness