Does RAG Know When Retrieval Is Wrong? Diagnosing Context Compliance under Knowledge Conflict

2026-05-14

AI Summary

This work addresses the vulnerability of Retrieval-Augmented Generation (RAG) systems to context hallucination when retrieved information conflicts with the model’s internal knowledge, a failure mode poorly captured by conventional evaluation metrics. To tackle this, the authors propose Context-Driven Decomposition (CDD), a novel framework that formalizes context adherence as a measurable structural dimension and introduces an intervention mechanism based on belief-decomposition probing to systematically diagnose RAG behavior under adversarial conditions. Through causal sensitivity analyses on Gemini and Claude model families—leveraging the Epi-Scale benchmark, TruthfulQA with injected misconceptions, and cross-model reruns—the study demonstrates that standard RAG achieves only 15.0% accuracy in such settings, whereas CDD elevates causal sensitivity to 64.1% for Gemini-2.5-Flash and sustains robustness at 71.3% and 69.9% under temporal drift and conflicting evidence scenarios, respectively.

Abstract

The Context-Compliance Regime in Retrieval-Augmented Generation (RAG) occurs when retrieved context dominates the final answer even when it conflicts with the model's parametric knowledge. Accuracy alone does not reveal how retrieved context causally shapes answers under such conflict. We introduce Context-Driven Decomposition (CDD), a belief-decomposition probe that operates at inference time and serves as an intervention mechanism for controlled retrieval conflict. Across Epi-Scale stress tests, TruthfulQA misconception injection, and cross- model reruns, CDD exposes three patterns. P1: context compliance is measurable in an upper-bound adversarial setting, where Standard RAG reaches 15.0% accuracy on TruthfulQA misconception injection (N=500). P2: adversarial accuracy gains transfer across model families: CDD improves accuracy on Gemini-2.5-Flash and on Claude Haiku/Sonnet/Opus, but rationale-answer causal coupling does not transfer. CDD reaches 64.1% mistake- injection causal sensitivity on Gemini-2.5-Flash, while sensitivities for all three Claude variants fall in the [-3%, +7%] range, suggesting that the Claude-side accuracy gains operate through a mechanism distinct from the explicit conflict-resolution trace. P3: explicit conflict decomposition improves robustness under temporal drift and noisy distractors, with CDD reaching 71.3% on temporal shifts and 69.9% on distractor evidence on the full Epi-Scale adversarial benchmark. These three patterns identify context-compliance as a structural axis along which standard RAG can be probed and intervened on, distinct from retrieval-quality or single-method robustness questions, and motivate releasing Epi-Scale for systematic study across model families and retrieval pipelines.

Problem

• Retrieval-Augmented Generation
• context compliance
• knowledge conflict
• adversarial evaluation
• RAG robustness

Innovation

• Context-Driven Decomposition
• Retrieval-Augmented Generation
• Context Compliance
• Knowledge Conflict
• Causal Intervention