Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability

2026-07-31

AI Summary

为了解决检索增强生成模型评估不一致的问题，提出Q-CARE框架，通过分解查询和答案并基于覆盖度与可验证性来提供细粒度的评估。

Abstract

Retrieval-augmented generation improves the factuality of large language models by grounding responses in retrieved evidence, yet existing evaluation frameworks struggle to provide consistent, fine-grained diagnostics across the diverse spectrum of user queries, ranging from close-ended fact-seeking to open-ended explanatory requests. We propose Q-CARE, a query-agnostic and fully reference-free framework that enables fine-grained assessment by decomposing queries into sub-queries and answers into atomic claims. Q-CARE establishes a unified evaluation principle based on query coverage and claim verifiability, yielding coverage-aware retriever metrics (C-Prec@k, C-nDCG@k) and claim-level generator metrics (Completeness, Conciseness, and Verifiableness). On a human-annotated benchmark spanning eight datasets, Q-CARE achieves higher correlation with human judgments than four existing RAG evaluation metrics, including RAGEval and RAGChecker, proving its effectiveness as a reliable, automated evaluation framework. Code and data are publicly available at https://github.com/DISL-Lab/Q-CaRE-COLM-26.

Problem

• retrieval-augmented generation
• evaluation framework
• query-agnostic
• fine-grained diagnostics
• user queries

Innovation

• query-agnostic
• reference-free
• fine-grained assessment
• query coverage
• claim verifiability