RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented Generation Systems

2025-06-01

AI Summary

Existing RAG evaluation frameworks overlook robustness under realistic noise, retrieval conflicts, and dynamically evolving facts. This paper introduces RARE, the first temporally aware benchmarking framework that jointly perturbs queries and documents. Its contributions are threefold: (1) RARE-Get, a knowledge-graph-driven automated pipeline for synthesizing multi-hop questions; (2) RARE-Met, a retrieval-conditioned robustness metric quantifying model resilience under systematic perturbations; and (3) RARE-Set, a dynamic, time-evolving benchmark comprising 400 timely, domain-specific documents and 48,322 questions. Methodologically, RARE integrates relation extraction, multi-hop question generation, temporal corpus modeling, and controlled perturbation injection. Experimental results reveal that RAG systems are most vulnerable to document-level perturbations—and multi-hop queries exhibit significantly lower robustness than single-hop ones—regardless of generator scale or architecture. These findings highlight critical gaps in current RAG robustness assessment and underscore the need for temporally grounded, joint-query-document evaluation.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Retrieval-Augmented Generation (RAG) enhances recency and factuality in answers. However, existing evaluations rarely test how well these systems cope with real-world noise, conflicting between internal and external retrieved contexts, or fast-changing facts. We introduce Retrieval-Aware Robustness Evaluation (RARE), a unified framework and large-scale benchmark that jointly stress-tests query and document perturbations over dynamic, time-sensitive corpora. One of the central features of RARE is a knowledge-graph-driven synthesis pipeline (RARE-Get) that automatically extracts single and multi-hop relations from the customized corpus and generates multi-level question sets without manual intervention. Leveraging this pipeline, we construct a dataset (RARE-Set) spanning 400 expert-level time-sensitive finance, economics, and policy documents and 48,322 questions whose distribution evolves as the underlying sources change. To quantify resilience, we formalize retrieval-conditioned robustness metrics (RARE-Met) that capture a model's ability to remain correct or recover when queries, documents, or real-world retrieval results are systematically altered. Our results show that RAG systems exhibit surprising vulnerability to perturbations, with document robustness consistently being the weakest point regardless of generator size or architecture. RAG systems consistently show lower robustness on multi-hop queries than single-hop queries across all domains.

Problem

• Evaluates RAG systems' resilience to real-world noise and conflicts
• Tests robustness against dynamic, time-sensitive data changes
• Measures vulnerability to query and document perturbations systematically

Innovation

• Knowledge-graph-driven synthesis pipeline for question generation
• Large-scale benchmark for dynamic, time-sensitive corpora
• Retrieval-conditioned robustness metrics for resilience quantification