From Business Requirements to Test Assertions: Evaluating LLM-Generated Oracles on Real Bugs

2026-07-11

AI Summary

This work addresses the oracle problem in automated testing by proposing a novel approach that leverages large language models (LLMs) to generate Java test oracles directly from natural language business requirements, without requiring source code or example input-output pairs. Using the Defects4J real-world defect dataset, the study employs differential analysis to extract behavioral changes and construct a structured gold standard (REQ) for systematic evaluation. It assesses the consistency of oracles generated by prominent LLMs—including DeepSeek-V3 and Llama-3—with both the specified requirements and the system under test. Experimental results demonstrate that LLM-generated oracles exhibit non-trivial generalization capability, showing significantly higher alignment with business requirements than with system implementation. Moreover, oracle accuracy shows no significant linear correlation with the technicality or ambiguity of the requirements.

Abstract

The oracle problem (determining the correct expected outcome for a test) remains a major bottleneck in automated testing, and is increasingly relevant as non-experts rely on AI-generated code they cannot reliably validate. We study whether large language models (LLMs) can generate generalizable test oracles directly from natural-language business requirements, without access to source code or example input-output pairs. We propose a reproducible, requirement-driven pipeline grounded in Defects4J. For each of 10 real bugs from Defects4J Lang (Bugs 1 and 3-11), we (i) extract behavioral changes via buggy/fixed diffs, (ii) manually translate the change into a business requirement, (iii) construct a requirement-derived oracle (REQ) as a gold standard, and (iv) prompt five LLMs (DeepSeek-V3, Gemma-3n, Llama-3, Mistral-7B, and Qwen-3) to generate Java oracle code. We evaluate oracle correctness and generalization under two targets: agreement with REQ and agreement with the system under test (SUT), reporting macro-averaged accuracy, precision, recall, and F1. LLMs achieve non-trivial generalization but with substantial bug- and model-level variance. Generated oracles align more closely with REQ than with SUT, and correlations between requirement technicality/ambiguity ratings and oracle accuracy are weak with wide confidence intervals. No detectable linear relationship exists between requirement properties and oracle accuracy in this dataset, suggesting that pretraining coverage and the semantic specificity of the required behavior dominate oracle correctness. As a pilot proof of concept, these findings are preliminary and are intended to establish feasibility and motivate larger-scale empirical investigation.

Problem

• oracle problem
• automated testing
• large language models
• test oracles
• business requirements

Innovation

• LLM-generated oracles
• requirement-driven testing
• oracle problem
• natural-language requirements
• automated testing