SATORI: Static Test Oracle Generation for REST APIs

2025-08-22

AI Summary

Existing REST API testing tools efficiently generate test data but exhibit weak static oracle capabilities, typically limited to crash detection, regression testing, and specification conformance checking. Method: We propose SATORI—the first approach to leverage large language models (LLMs) for black-box static oracle generation. Given OpenAPI specifications, SATORI automatically infers expected behaviors from response field names and semantic descriptions, and synthesizes executable assertions integrated into PostmanAssertify. Contribution/Results: SATORI complements dynamic oracles and significantly improves oracle coverage. Evaluated on 12 industrial APIs, it generates hundreds of valid oracles per operation, achieving an F1-score of 74.3%—surpassing state-of-the-art dynamic methods. It uncovered 18 real-world defects, prompting documentation corrections and establishing a novel paradigm for REST API automated testing.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

REST API test case generation tools are evolving rapidly, with growing capabilities for the automated generation of complex tests. However, despite their strengths in test data generation, these tools are constrained by the types of test oracles they support, often limited to crashes, regressions, and noncompliance with API specifications or design standards. This paper introduces SATORI (Static API Test ORacle Inference), a black-box approach for generating test oracles for REST APIs by analyzing their OpenAPI Specification. SATORI uses large language models to infer the expected behavior of an API by analyzing the properties of the response fields of its operations, such as their name and descriptions. To foster its adoption, we extended the PostmanAssertify tool to automatically convert the test oracles reported by SATORI into executable assertions. Evaluation results on 17 operations from 12 industrial APIs show that SATORI can automatically generate up to hundreds of valid test oracles per operation. SATORI achieved an F1-score of 74.3%, outperforming the state-of-the-art dynamic approach AGORA+ (69.3%)-which requires executing the API-when generating comparable oracle types. Moreover, our findings show that static and dynamic oracle inference methods are complementary: together, SATORI and AGORA+ found 90% of the oracles in our annotated ground-truth dataset. Notably, SATORI uncovered 18 bugs in popular APIs (Amadeus Hotel, Deutschebahn, FDIC, GitLab, Marvel, OMDb and Vimeo) leading to documentation updates by the API maintainers.

Problem

• Generating test oracles for REST APIs automatically
• Overcoming limitations in current oracle types supported
• Inferring expected API behavior from OpenAPI specifications

Innovation

• Uses LLMs to infer API behavior statically
• Generates test oracles from OpenAPI specifications
• Converts oracles to executable Postman assertions