Quantifying the Robustness of Retrieval-Augmented Language Models Against Spurious Features in Grounding Data

2025-03-07

AI Summary

This work addresses the robustness of large language models (LLMs) in Retrieval-Augmented Generation (RAG) systems against semantically irrelevant spurious features—such as formatting, layout, and metadata—that constitute implicit noise, thereby filling a critical gap in prior research focused solely on explicit semantic noise. We propose the first taxonomy of spurious features tailored to RAG, and through controlled causal experiments and multi-model attribution analysis, we empirically establish—statistically and for the first time—their prevalence and substantial impact on model behavior. Notably, we discover that certain spurious features can be leveraged beneficially, challenging the conventional assumption that spuriousness inherently implies harm. To support systematic investigation, we release an open-source benchmark comprising a curated dataset and evaluation framework enabling spurious feature injection, quantitative robustness assessment, and fine-grained diagnostic analysis—thereby advancing standardization and reproducibility in RAG robustness research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Robustness has become a critical attribute for the deployment of RAG systems in real-world applications. Existing research focuses on robustness to explicit noise (e.g., document semantics) but overlooks spurious features (a.k.a. implicit noise). While previous works have explored spurious features in LLMs, they are limited to specific features (e.g., formats) and narrow scenarios (e.g., ICL). In this work, we statistically confirm the presence of spurious features in the RAG paradigm, a robustness problem caused by the sensitivity of LLMs to semantic-agnostic features. Moreover, we provide a comprehensive taxonomy of spurious features and empirically quantify their impact through controlled experiments. Further analysis reveals that not all spurious features are harmful and they can even be beneficial sometimes. Extensive evaluation results across multiple LLMs suggest that spurious features are a widespread and challenging problem in the field of RAG. The code and dataset will be released to facilitate future research. We release all codes and data at: $\href{https://github.com/maybenotime/RAG-SpuriousFeatures}{https://github.com/maybenotime/RAG-SpuriousFeatures}$.

Problem

• Assessing RAG model robustness against spurious features.
• Identifying and categorizing spurious features in RAG systems.
• Quantifying impact of spurious features on RAG performance.

Innovation

• Identifies spurious features in RAG systems
• Develops taxonomy for spurious features impact
• Releases code and dataset for future research