Magic Mushroom: A Customizable Benchmark for Fine-grained Analysis of Retrieval Noise Erosion in RAG Systems

2025-06-04

AI Summary

Existing RAG benchmarks fail to emulate the complex, heterogeneous noise distributions prevalent in real-world retrieval, leading to unreliable robustness evaluation. Method: We introduce Magic Mushroom—the first configurable RAG benchmark—systematically modeling four empirically grounded noise types, with emphasis on “toxic mushroom” contexts that appear relevant but are actually misleading. We propose a novel four-dimensional noise spectrum, enabling controllable injection of noise type, intensity, and composition across single- and multi-hop QA scenarios. Our approach integrates linguistic feature modeling, human–LLM collaborative construction of high-quality noisy samples, and a modular architecture with a cross-scale evaluation framework—including LLM-based generators and classical denoising strategies. Contribution/Results: Experiments reveal that mainstream models exhibit high sensitivity to noise distribution, suffering nonlinear, cliff-like performance degradation. The benchmark is open-sourced, comprising 7,468 single-hop and 3,925 multi-hop QA pairs, enabling fine-grained robustness analysis.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLMs) by incorporating external retrieved information, mitigating issues such as hallucination and outdated knowledge. However, RAG systems are highly sensitive to retrieval noise prevalent in real-world scenarios. Existing benchmarks fail to emulate the complex and heterogeneous noise distributions encountered in real-world retrieval environments, undermining reliable robustness assessment. In this paper, we define four categories of retrieval noise based on linguistic properties and noise characteristics, aiming to reflect the heterogeneity of noise in real-world scenarios. Building on this, we introduce Magic Mushroom, a benchmark for replicating"magic mushroom"noise: contexts that appear relevant on the surface but covertly mislead RAG systems. Magic Mushroom comprises 7,468 single-hop and 3,925 multi-hop question-answer pairs. More importantly, Magic Mushroom enables researchers to flexibly configure combinations of retrieval noise according to specific research objectives or application scenarios, allowing for highly controlled evaluation setups. We evaluate LLM generators of varying parameter scales and classic RAG denoising strategies under diverse noise distributions to investigate their performance dynamics during progressive noise encroachment. Our analysis reveals that both generators and denoising strategies have significant room for improvement and exhibit extreme sensitivity to noise distributions. Magic Mushroom emerges as a promising tool for evaluating and advancing noise-robust RAG systems, accelerating their widespread deployment in real-world applications. The Magic Mushroom benchmark is available at the https://drive.google.com/file/d/1aP5kyPuk4L-L_uoI6T9UhxuTyt8oMqjT/view?usp=sharing.

Problem

• RAG systems are sensitive to real-world retrieval noise
• Existing benchmarks lack realistic noise distribution emulation
• Magic Mushroom enables customizable noise configuration for evaluation

Innovation

• Customizable benchmark for RAG noise analysis
• Defines four categories of retrieval noise
• Flexible noise configuration for evaluation