Requirements-Augmented Generation for Trustworthy Acceptance Testing of LLM-Based Software

2026-08-13

AI Summary

This work addresses the challenge of unreliable test oracles in acceptance testing of large language model–based software (LBS), which stems from context dependency and output stochasticity. The authors propose the first automated acceptance testing framework for LBS, featuring a Requirement-Enhanced Augmented Generation (REAG) approach that integrates user requirements, domain knowledge, and user personas to produce intent-aware, context-sensitive test oracles. To ensure oracle reliability, they further introduce a cascaded decision mechanism calibrated by simulated expert consensus, combining adaptive retrieval-augmented generation with conformal risk control. Evaluated on an industrial nutrition counseling case study, REAG-generated oracles achieved a quality score of 3.91/5 (with 82% of cases rated acceptable or borderline), while the cascaded mechanism boosted accuracy to 98.8%, improved oracle quality to 4.30, and reduced testing costs by 31.7%.

Abstract

LLM-based software (LBS) integrates large language models as core components to deliver flexible, personalised responses. Unlike traditional software with deterministic outputs, LBSs exhibit context-dependent, stochastic behaviour that renders classical acceptance testing and test oracles insufficient: the same query may require fundamentally different responses depending on user personas and software context. This gap creates an urgent need for automated acceptance testing frameworks that autonomously interpret user instructions, while reliably inferring user intentions in a changing environment. In this paper, we present an automated acceptance testing framework for LBS with calibrated verdict reliability via two technical contributions. First, we introduce Requirements-Augmented Generation (REAG), which interprets user intentions by retrieving relevant software requirements, domain knowledge, and personas via adaptive RAG and self-reasoning to generate context-aware test oracles. Second, recognising that oracle generation may retrieve irrelevant constraints, misinterpret intent, or hallucinate requirements, we introduce a confidence-calibrated cascade judgment. This method quantifies verdict reliability via simulated expert agreement -- accepting high-confidence verdicts, escalating ambiguous cases, or abstaining when uncertain -- with empirical reliability guarantees backed by conformal risk control. An industrial case study on a production nutrition advisory application demonstrates that REAG achieves a 3.91/5 oracle quality score, reaching qualified or marginal oracle quality in 82% of cases. The confidence-calibrated cascade achieves 98.8% accuracy, improves oracle quality from 3.91 to 4.30 by filtering unqualified outputs, and delivers a 31.7% cost-efficiency improvement over single-judge baselines, validating industrial viability

Problem

• LLM-based software
• acceptance testing
• test oracles
• user intention
• context-dependent behavior

Innovation

• Requirements-Augmented Generation
• Confidence-Calibrated Cascade
• Adaptive RAG
• Conformal Risk Control
• Test Oracle for LLM