Purifying Multimodal Retrieval: Fragment-Level Evidence Selection for RAG

2026-04-30

AI Summary

Existing multimodal retrieval-augmented generation (MRAG) approaches treat entire documents as evidence, introducing substantial irrelevant noise that degrades generation quality. To address this limitation, this work proposes FES-RAG, a fine-grained evidence selection framework that decomposes documents into textual sentences and visual region patches. It introduces a novel Fragment Information Gain metric to precisely identify the most supportive fragments for the generation task. Integrating multimodal large language models with knowledge distillation, FES-RAG constructs a lightweight selector that significantly enhances factual accuracy and coherence while reducing context length and computational overhead. On the M2RAG benchmark, the method achieves up to a 27% improvement in CIDEr score over document-level baselines.

Abstract

Multimodal Retrieval-Augmented Generation (MRAG) is widely adopted for Multimodal Large Language Models (MLLMs) with external evidence to reduce hallucinations. Despite its success, most existing MRAG frameworks treat retrieved evidence as indivisible documents, implicitly assuming that all content within a document is equally informative. In practice, however, sometimes only a small fraction of a document is relevant to a given query, while the remaining content introduces substantial noise that may lead to performance degradation. We address this fundamental limitation by reframing MRAG as a fine-grained evidence selection problem. We propose Fragment-level Evidence Selection for RAG (FES-RAG), a framework that selects atomic multimodal fragments rather than entire documents as grounding evidence. FES-RAG decomposes retrieved multimodal documents into sentence-level textual fragments and region-level visual fragments, enabling precise identification of evidence that directly supports generation. To guide fragment selection, we introduce Fragment Information Gain (FIG), a principled metric that measures the marginal contribution of each fragment to the MLLM's generation confidence. Based on FIG, we distill fragment-level utility judgments from a high-capacity MLLM into a lightweight selector, achieving accurate evidence selection with low inference overhead. Experiments on the M2RAG benchmark show that FES-RAG consistently outperforms state-of-the-art document-level MRAG methods, achieving up to 27 percent relative improvement in CIDEr. By selecting fewer yet more informative fragments, our approach substantially reduces context length while improving factual accuracy and generation coherence.

Problem

• Multimodal Retrieval-Augmented Generation
• evidence selection
• hallucination reduction
• retrieval noise
• fine-grained grounding

Innovation

• Fragment-level Evidence Selection
• Multimodal Retrieval-Augmented Generation
• Fragment Information Gain
• Evidence Distillation
• Multimodal Large Language Models