Combining Static and Dynamic Approaches for Mining and Testing Constraints for RESTful API Testing

2025-04-24

AI Summary

Existing dynamic analysis techniques for RESTful API testing incompletely mine response-body logical constraints due to insufficient input diversity, leading to constraint underestimation. Method: This paper proposes the first hybrid constraint-mining framework integrating static analysis (OpenAPI specification parsing) and dynamic analysis (runtime execution profiling), augmented by a novel LLM-based Observation-Confirmation (OC) prompting mechanism to mitigate hallucination. It enables synergistic validation and enhancement between static and dynamic constraints. Contribution/Results: The LLM+OC approach achieves 91.2% constraint identification accuracy; the integrated tool RBCTest attains 78.5% overall precision. Our method uncovers 107 constraints missed by dynamic-only methods and refines 46 constraints with higher precision. Furthermore, it detects 21 specification-implementation inconsistencies across eight real-world APIs—four of which have been confirmed by developer communities.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: Safety and Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

In API testing, deriving logical constraints on API response bodies is crucial in generating the test cases to cover various aspects of RESTful APIs. However, existing approaches are limited to dynamic analysis in which constraints are extracted from the execution of APIs as part of the system under test. The key limitation of such a dynamic approach is its under-estimation in which inputs in API executions are not sufficiently diverse to uncover actual constraints on API response bodies. In this paper, we propose to combine a novel static analysis approach (in which the constraints for API response bodies are mined from API specifications), with the dynamic approach (which relies on API execution data). We leverage large language models (LLMs) to comprehend the API specifications, mine constraints for response bodies, and generate test cases. To reduce LLMs' hallucination, we apply an Observation-Confirmation (OC) scheme which uses initial prompts to contextualize constraints. %, allowing subsequent prompts to more accurately confirm their presence. Our empirical results show that~LLMs with OC prompting achieve high precision in constraint mining with the average of 91.2%. When combining static and dynamic analysis, our tool, RBCTest , achieves a precision of 78.5%. RBCTest detects 107 constraints that the dynamic approach misses and 46 more precise constraints. We also use its generated test cases to detect 21 mismatches between the API specification and actual response data for 8 real-world APIs. Four of the mismatches were, in fact, reported in developers' forums.

Problem

• Mining constraints from API specs and execution data
• Improving test case diversity via static-dynamic analysis
• Reducing LLM hallucination in constraint mining precision

Innovation

• Combines static and dynamic analysis for API testing
• Uses LLMs to mine constraints from API specifications
• Applies OC scheme to reduce LLM hallucination