HoH: A Dynamic Benchmark for Evaluating the Impact of Outdated Information on Retrieval-Augmented Generation

2025-03-03

AI Summary

This work addresses the problem of outdated information in retrieval-augmented generation (RAG) knowledge bases, which degrades response accuracy and introduces harmful outputs. To systematically expose the dual impact of obsolescence on RAG accuracy and safety, we introduce HoH—the first dynamic temporal benchmark for RAG. Methodologically, we propose the first analytical framework focused on the *outdated interference mechanism*, and design a temporal QA data generation method that synergistically combines token-level diff analysis with large language models, integrating temporal knowledge modeling and diagnostic RAG evaluation. Empirical results demonstrate that state-of-the-art RAG systems exhibit severe vulnerability to outdated information in both retrieval and generation stages; such obsolescence significantly reduces answer accuracy and triggers factual hallucinations as well as unsafe content. This work establishes foundational benchmarks and methodologies for advancing robustness and trustworthiness in RAG systems.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

While Retrieval-Augmented Generation (RAG) has emerged as an effective approach for addressing the knowledge outdating problem in Large Language Models (LLMs), it faces a critical challenge: the prevalence of outdated information in knowledge bases. Current research primarily focuses on incorporating up-to-date information, yet the impact of outdated information coexisting in retrieval sources remains inadequately addressed. To bridge this gap, we introduce HoH, the first benchmark specifically designed to evaluate the impact of outdated information on RAG. Our benchmark leverages token-level diff algorithms combined with LLM pipelines to efficiently create a large-scale QA dataset that accurately captures temporal knowledge evolution in real-world facts. Through comprehensive experiments, we reveal that outdated information significantly degrades RAG performance in two critical ways: (1) it substantially reduces response accuracy by distracting models from correct information, and (2) it can mislead models into generating potentially harmful outputs, even when current information is available. Current RAG approaches struggle with both retrieval and generation aspects when handling outdated information. These findings highlight the urgent need for innovative solutions to address the temporal challenges in RAG.

Problem

• Evaluates outdated information impact on Retrieval-Augmented Generation (RAG).
• Creates benchmark to assess temporal knowledge evolution in RAG.
• Reveals outdated info reduces accuracy and risks harmful outputs.

Innovation

• HoH benchmark evaluates outdated information impact
• Token-level diff algorithms create temporal QA dataset
• Reveals outdated info reduces RAG accuracy, misleads models