test oracle design

This work addresses the challenge of efficiently testing black-box systems with side effects by proposing a test generation approach that integrates under-approximate typing with effect systems. The method employs symbolic traces to capture data and control dependencies of side-effecting operations, preserving essential constraints to guide test case synthesis. Precise coverage is achieved through an integration of property-based testing and model checking. The implemented tool, Clouseau, demonstrates substantial improvements over default strategies in frameworks such as QCheck and P, achieving test effectiveness comparable to state-of-the-art hand-crafted test suites. These results validate both the efficacy and practicality of the proposed methodology.

This work addresses the limitations of traditional black-box API testing, which struggles to generate effective state-dependent call sequences due to a lack of behavioral semantics and suffers from the test oracle problem. The authors propose a model-checking-based systematic testing approach that employs TLA+ to formally model API state evolution and leverages the TLC model checker to perform coverage-guided, breadth-first traversal of the state space. This method mitigates state explosion while producing test sequences with provable coverage guarantees over the behavioral model. To enhance runtime validation, the approach integrates Glacier, an executable first-order logic contract language. Empirical evaluation on the EvoMaster benchmark demonstrates complete state coverage, effective detection of multi-operation interaction bugs, and confirms the method’s scalability and practical applicability boundaries.

To address the challenges of low accuracy and difficulty in automated test oracle generation for non-regression testing, this paper proposes a multi-agent collaborative deliberation framework. The method integrates reasoning-based generation, sandboxed execution validation, and an iterative self-refinement mechanism guided by runtime error feedback, establishing a structured deliberation process that ensures execution-grounded and progressively optimized oracles. Key innovations include: (1) a role-specialized multi-agent architecture, (2) a verifiable candidate implementation execution mechanism, and (3) a closed-loop self-correction algorithm. Evaluated on seven benchmarks, the approach significantly outperforms existing methods: GPT-4.1-Mini achieves 57.73% oracle accuracy on LiveCodeBench (+11.43% absolute gain), 95.45% defect detection rate on HumanEval, and 69.32% program repair success rate.

Protocol testing faces two key challenges: (1) generating high-quality inputs—syntactically and semantically valid messages that cover diverse behavioral scenarios—and (2) reliably verifying outputs due to the absence of effective test oracles. To address these, this paper introduces I/O Grammar, the first formalism unifying message syntax, state transitions, and interactive behavior across multiple protocol roles (e.g., client and server), enabling systematic test generation, mock-based simulation, and response validation. Built upon the FANDANGO framework, our approach integrates user-defined constraints with a k-path guided exploration strategy, establishing a cohesive tripartite framework encompassing test generation, mocking, and oracle-free response judgment. Experimental evaluation on DNS, FTP, and SMTP protocols demonstrates that our method significantly outperforms random testing in both input-space and response-space coverage efficiency, achieving faster and more comprehensive functional coverage.

Traditional model-based testing (MBT) for protocols relies on manual RFC parsing to construct behavioral models, resulting in high modeling overhead and hindering practical adoption. This paper proposes an LLM-driven automated black-box testing framework that extracts protocol behavioral specifications from unstructured sources—including RFCs and technical blogs—using large language models, and integrates symbolic execution to generate semantically complete test cases. It introduces the first unified approach jointly performing behavioral modeling and oracle synthesis, enabling semantic-level protocol testing without manual modeling. Evaluated on DNS, the framework uncovered 26 real-world defects across 10 mainstream DNS implementations, including 11 previously unknown vulnerabilities. This significantly advances automation in protocol conformance testing and enhances vulnerability detection capability.

This work addresses the limitations of existing large language models in generating concurrent, stateful Rust API tests—namely, frequent violations of preconditions, insufficient depth, and degeneration into sequential execution—as well as the heavy manual modeling burden of traditional model-driven testing. The authors propose a Petri net–guided test generation framework that employs colored Petri nets to formally model API resources, lifecycles, and causal dependencies. This formalism yields valid, near-valid, and partially ordered concurrent scenarios as constrained intermediate representations. Guided by local fidelity contracts, structural repair loops, and schedule shaping mechanisms, large language models synthesize executable tests exhibiting high conflict and coverage. The approach enables low-cost translation from formal scenarios to test code, significantly improving the legality, depth, and concurrency coverage of generated tests.

This work addresses the high cost of manually writing formal specifications and the limitations of existing large language model (LLM)-based approaches that require white-box access to source code, thereby posing intellectual property and deployment constraints. The authors propose a black-box-driven method that leverages only test code and dynamic execution traces to generate candidate Java Modeling Language (JML) specifications via an LLM. These candidates are locally validated using bounded model checking, and an iterative feedback loop refines them based on verification outcomes. This approach is the first to enable fully automated formal specification generation without any access to the program’s internal structure. Evaluated on the SpecGenBench benchmark, it demonstrates that test-derived information effectively guides specification synthesis, while also highlighting critical challenges in checker compatibility and diagnostic feedback, substantially enhancing industrial applicability.

This work addresses the limitation of conventional execution coverage in UI component testing, which fails to verify whether tests adequately capture behavioral relationships implied by APIs and documentation. The paper proposes the first evaluation framework based on inferred metamorphic relations (MRs): it automatically derives MRs using a UI-specific taxonomy from source code and documentation, aligns test executions to these MRs through deterministic and semantic analysis, and introduces relation-level MR coverage as a novel metric. By treating inferred MRs as empirical benchmarks for behavioral validation, the approach exposes verification gaps invisible to traditional coverage metrics—particularly in weak-oracle scenarios. Empirical results across three LLM configurations show MR coverage ranging only from 42.5% to 47.6%, substantially lower than MR reachability; uncovered MRs are predominantly of the weak-oracle type, demonstrating that MR coverage meaningfully complements conventional metrics and offers practical utility in fault detection and issue mapping.

Existing symbolic execution tools, such as Java Ranger which supports path merging, lack formal verification of their correctness, potentially introducing unsoundness through optimizations and thereby weakening program verification guarantees. This work presents the first comprehensive formal semantic framework for Java symbolic execution with path merging, systematically transforming programs into logical expressions through a sequence of semantics-preserving code transformations. It rigorously establishes the soundness of Large-Block Encoding under a simplified Java semantics, proving that this optimization faithfully preserves program behavior. By formally ensuring that path merging and related optimizations do not introduce semantic deviations, this research significantly enhances the trustworthiness of symbolic execution tools in safety-critical systems.

This work addresses a critical limitation in existing large language model–based test oracle generation methods, which rely on iterative self-repair and execution feedback and often fall into a “self-repair trap,” producing assertions that pass yet fail to expose real faults. To overcome this, the paper proposes DCAware, a novel framework that introduces, for the first time, a dual-context awareness mechanism. This mechanism non-iteratively integrates structured static program context with selectively retrieved dynamic execution states to achieve high signal-to-noise ratio context awareness. By abandoning the conventional multi-round repair paradigm, DCAware simultaneously maintains high execution success rates while significantly enhancing both fault-revealing capability and computational efficiency, outperforming current state-of-the-art approaches across the board.