The RAT: A Unified Bayesian Model for RAG Evaluation

2026-08-25

AI Summary

本文提出一种贝叶斯模型来评估检索增强生成系统，通过联合建模检索成功率、回避行为和答案正确性，区分任务成功与生成器行为，揭示了系统间的行为差异。

Abstract

Evaluating Retrieval-Augmented Generation (RAG) systems requires assessing not only end-to-end correctness but also how individual components interact and how errors propagate through the pipeline. We introduce a Bayesian evaluation framework that jointly models retrieval success, abstention behavior, and answer correctness, factorized according to the pipeline's information flow. The model distinguishes task success. Whether the user received a correct answer (from generator success) and whether the generator behaved appropriately given the retrieval outcome. We apply the framework to 27 RAG configurations across three datasets, three retrievers, and three generators, and show that the conditional decomposition reveals substantial behavioral differences between systems that appear equivalent under marginal metrics. We further analyze the annotation allocation problem, demonstrating that retrieval-success annotations are more informative than task-success annotations for estimating policy adherence, and provide an information-theoretic explanation for this asymmetry. Finally, we extend the model to incorporate LLM-as-a-judge annotations as calibrated noisy observations, enabling practitioners to combine limited human judgments with cheaper automated assessments within a unified probabilistic model.

Problem

• Retrieval-Augmented Generation
• evaluation framework
• retrieval success
• answer correctness
• behavioral differences

Innovation

• Bayesian evaluation framework
• retrieval success
• answer correctness
• conditional decomposition
• LLM-as-a-judge