BOSQTGEN: Breaking the Sound Barrier in Test Generation

2025-10-22

AI Summary

Incomplete API contracts impede black-box testing in polyglot systems, particularly hindering the generation of structured, semantically valid inputs when source code is inaccessible. To address this, we propose an LLM-driven conformance testing method that decomposes OpenAPI specifications into composable semantic primitives, leverages large language models to synthesize hierarchical and semantically coherent inputs, and integrates combinatorial testing to enhance sampling efficiency—overcoming redundancy inherent in random generation. Our approach is source-code-agnostic, language-agnostic, and enables automated, structured test case synthesis for RESTful APIs. Evaluated on standard benchmarks, it achieves an average 82% code coverage—surpassing the best prior tools by over 20 percentage points—and approaches the quality of manually authored tests.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Modern software is increasingly built by composing APIs, elevating the API contract to a critical role. Inadequate contracts, however, lead to mismatched expectations and failures, creating a pressing need for robust conformance testing. Current test generation techniques are hindered by key challenges: polyglot systems, source code inaccessibility, a cost-reliability trade-off, and, most critically, the difficulty of generating structured inputs. We introduce BOSQTGEN, a novel black-box methodology and tool for API test generation. BOSQTGEN utilizes a novel approach for decomposing API specifications into primitives, using LLMs to suggest coherent strata for them, and employing combinatorial testing to efficiently sample over these values. This approach ensures coverage of critical interactions while avoiding the redundancy of random sampling. The resulting BOSQTGEN system achieves an average of 82% code coverage on RESTful benchmarks, often a 20% or more increase over prior state-of-the-art systems and nearing parity with hand-written test suites. Providing a fully API-driven approach to test generation, enables developers to automatically create high-quality test cases for validation or test-driven development.

Problem

• Generating structured inputs for API testing
• Overcoming polyglot systems and source inaccessibility
• Balancing cost-reliability trade-off in test generation

Innovation

• Decomposes API specifications into primitives for testing
• Uses LLMs to suggest coherent strata for values
• Employs combinatorial testing to efficiently sample values