REST API Testing in DevOps: A Study on an Evolving Healthcare IoT Application

2024-10-16

arXiv.org

AI Summary

In DevOps-enabled healthcare IoT systems, the continuous evolution of REST APIs undermines regression testing effectiveness, posing critical quality and safety risks. Method: This study conducts the first systematic empirical evaluation of regression testing capabilities of five state-of-the-art tools—RESTest, EvoMaster, Schemathesis, RESTler, and RestTestGen—in a real-world healthcare IoT setting. The evaluation spans 17 APIs, 120 endpoints, and 14 release iterations, integrating contract-based validation, fuzzing, and evolution-aware strategies. Contribution/Results: The experiments uncovered 18 previously unknown defects and 23 regression faults, achieving up to 84% endpoint coverage—yet with an average test overhead exceeding 80%. This work establishes the first empirical benchmark for regression testing in healthcare IoT DevOps and empirically characterizes the practical trade-offs among fault detection, coverage, and resource cost.

Technology Category

• Application Domains: Internet of Things, Sensor Networks & Smart Cities
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Constraint Satisfaction and Optimization: Solvers and Tools

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Healthcare Internet of Things (IoT) applications often integrate various third-party healthcare applications and medical devices through REST APIs, resulting in complex and interdependent networks of REST APIs. Oslo City's healthcare department collaborates with various industry partners to develop such healthcare IoT applications enriched with a diverse set of REST APIs. Following the DevOps process, these REST APIs continuously evolve to accommodate evolving needs such as new features, services, and devices. Oslo City's primary goal is to utilize automated solutions for continuous testing of these REST APIs at each evolution stage, thereby ensuring their dependability. Although the literature offers various automated REST API testing tools, their effectiveness in regression testing of the evolving REST APIs of healthcare IoT applications within a DevOps context remains undetermined. This paper evaluates state-of-the-art and well-established REST API testing tools-specifically, RESTest, EvoMaster, Schemathesis, RESTler, and RestTestGen-for the regression testing of a real-world healthcare IoT application, considering failures, faults, coverage, regressions, and cost. We conducted experiments using all accessible REST APIs (17 APIs with 120 endpoints), and 14 releases evolved during DevOps. Overall, all tools generated tests leading to several failures, 18 potential faults, up to 84% coverage, 23 regressions, and over 80% cost overhead.

Problem

• Evaluating REST API testing tools for healthcare IoT applications
• Assessing tool effectiveness in DevOps regression testing
• Identifying failures and faults in evolving REST APIs

Innovation

• Automated REST API testing in DevOps
• Evaluating five REST API testing tools
• Testing 17 APIs with 120 endpoints