Test Amplification for REST APIs Using"Out-of-the-box"Large Language Models

2025-03-13

AI Summary

To address insufficient boundary-value coverage and poor test-case readability in REST API automated testing, this paper proposes an API test-case augmentation method that leverages off-the-shelf commercial large language models (LLMs)—specifically ChatGPT and GitHub Copilot—without fine-tuning. Our approach employs test-quality-oriented prompt engineering to systematically validate LLMs’ effectiveness in generating protocol-compliant boundary values, yielding 12 reusable, empirically grounded prompt design principles. Experimental evaluation demonstrates that the generated tests significantly improve path and parameter boundary coverage while maintaining high semantic clarity and human interpretability. This work constitutes the first systematic empirical validation of zero-shot LLMs for practical API test augmentation, establishing a novel paradigm for low-barrier, high-quality API testing.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

REST APIs are an indispensable building block in today's cloud-native applications, so testing them is critically important. However, writing automated tests for such REST APIs is challenging because one needs strong and readable tests that exercise the boundary values of the protocol embedded in the REST API. In this paper, we report our experience with using"out of the box"large language models (ChatGPT and GitHub's Copilot) to amplify REST API test suites. We compare the resulting tests based on coverage and understandability, and we derive a series of guidelines and lessons learned concerning the prompts that result in the strongest test suite.

Problem

• Challenges in writing automated tests for REST APIs
• Using large language models to amplify REST API test suites
• Comparing test coverage and understandability from generated tests

Innovation

• Utilizes large language models for test amplification
• Focuses on REST API boundary value testing
• Evaluates test coverage and understandability