🤖 AI Summary
This work addresses the lack of reliable end-to-end testing methods for Retrieval-Augmented Generation (RAG) systems, which stems from complex interactions among their components. The authors propose RagTester, the first automated evaluation framework that introduces a coverage-guided test generation strategy. RagTester systematically synthesizes test documents, queries, and expected answers, and leverages large language models as judges to detect failure modes such as inaccurate retrieval, unsupported answers, and insufficient context utilization. Evaluated across 24 configurations with 72,000 tests, RagTester identified 21,633 faults—6.6% more than baseline methods—and outperformed baselines in 20 out of 24 configurations, substantially enhancing pre-deployment validation capabilities for RAG systems.
📝 Abstract
Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to use external and domain-specific knowledge, but its reliability depends on the interaction between the generative model, embedding model, retrieval mechanism, and prompt construction strategy. We present RagTester, an automated end-to-end testing approach for RAG systems. RagTester generates retrieval documents, test inputs, and expected outputs; executes the tests; and evaluates the resulting answers using an LLM as a judge. Its test-generation strategy targets complex passages, unsupported queries, and document-coverage criteria. We evaluate RagTester using eight LLMs and six embedding models, yielding 24 compatible configurations, and compare it with a baseline test-input generator. Across 72,000 test executions, RagTester detected 21,633 failures, 6.6% more than the baseline, and outperformed it in 20 of the 24 configurations. The detected failures include inaccurate retrieval, unsupported answers, incomplete use of retrieved context, and difficulties interpreting complex passages. These results show that coverage-oriented test generation can effectively expose failures caused by the interaction between retrieval and generation components and support the assessment of RAG configurations before deployment.