Trust Before Fusion: QIMG-7 and Source-Aware Resolution for Polluted Multimodal RAG

2026-07-12

AI Summary

This work addresses the vulnerability of multimodal retrieval-augmented generation (RAG) to factual errors when exposed to contaminated retrieval content, such as fabricated text or misleading images. To systematically quantify this issue, the authors introduce the QIMG-7 benchmark and propose Source-Aware Trust Resolution (SATR), a training-free method that dynamically selects among parametric, text-only, or full multimodal answers to ensure reliable responses. The Field-Selector variant of SATR achieves a balanced score of 0.816, outperforming full multimodal fusion by 11.7 points and significantly surpassing baseline approaches like cascaded routers. These results underscore the critical importance of explicitly modeling textual trustworthiness in text-priority scenarios.

Abstract

Multimodal retrieval-augmented generation (RAG) is often evaluated with clean evidence, yet real retrieval can return topically relevant but unreliable content: false text and misleading images from corrupted metadata, entity swaps, typographic overlays, semantic edits, adversarial patches, blends, or style transfer. We introduce QIMG-7, a controlled benchmark for multimodal retrieval pollution in multi-sentence factual QA, spanning four datasets, seven image-attack families, and 16 paired clean/polluted regimes, for 1,760 evaluation rows per method. Across four generator/gate stacks, naive multimodal fusion is brittle: in the main gpt-4o-mini stack, Full-MM support drops from 0.908 with clean text to 0.490 with polluted text, often making Parametric fallback safer than retrieval. We propose source-aware trust resolution (SATR), a training-free approach that compares Parametric, Text-only, and Full-MM candidate answers and selects among candidate answers or falls back based on source reliability. The Field-Selector variant achieves the best balanced score, 0.816, improving over Full-MM by 11.7 points and over the Cascaded Router by 2.7 points. Ablations show that, in this text-first setting, explicit text-reliability modeling is the dominant driver of these gains. Overall, in text-first factual QA with multimodal retrieval conflict, our results support selective trust rather than unconditional fusion. Artifacts are available at https://github.com/SaadElDine/Trust_Before_Fusion.

Problem

• multimodal retrieval-augmented generation
• retrieval pollution
• unreliable multimodal evidence
• factual question answering
• trust resolution

Innovation

• multimodal RAG
• retrieval pollution
• source-aware trust resolution
• QIMG-7 benchmark
• trust before fusion