MEG-RAG: Quantifying Multi-modal Evidence Grounding for Evidence Selection in RAG

2026-04-27

AI Summary

This work addresses the limitation of existing multimodal retrieval-augmented generation (MRAG) systems in accurately assessing whether retrieved evidence genuinely supports the semantic core of the generated answer, often relying on heuristic positional confidence while overlooking information density. To this end, the paper introduces Multimodal Evidence Grounding (MEG), a novel metric that anchors high-IDF, information-rich tokens through semantic certainty, and proposes the MEG-RAG framework to train a multimodal re-ranker that precisely aligns retrieval evidence with semantic anchor points in the answer. This approach pioneers a semantics-aware mechanism for quantifying multimodal evidence, eschewing conventional probabilistic confidence in favor of high-value semantic content. Experiments demonstrate that MEG-RAG significantly outperforms strong baselines on the M²RAG benchmark and exhibits robust generalization across diverse teacher models.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multimodal Retrieval-Augmented Generation (MRAG) addresses key limitations of Multimodal Large Language Models (MLLMs), such as hallucination and outdated knowledge. However, current MRAG systems struggle to distinguish whether retrieved multimodal data truly supports the semantic core of an answer or merely provides superficial relevance. Existing metrics often rely on heuristic position-based confidence, which fails to capture the informational density of multimodal entities. To address this, we propose Multi-modal Evidence Grounding (MEG), a semantic-aware metric that quantifies the contribution of retrieved evidence. Unlike standard confidence measures, MEG utilizes Semantic Certainty Anchoring, focusing on high-IDF information-bearing tokens that better capture the semantic core of the answer. Building on MEG, we introduce MEG-RAG, a framework that trains a multimodal reranker to align retrieved evidence with the semantic anchors of the ground truth. By prioritizing high-value content based on semantic grounding rather than token probability distributions, MEG-RAG improves the accuracy and multimodal consistency of generated outputs. Extensive experiments on the M$^2$RAG benchmark show that MEG-RAG consistently outperforms strong baselines and demonstrates robust generalization across different teacher models.

Problem

• Multimodal Retrieval-Augmented Generation
• evidence grounding
• semantic relevance
• hallucination
• information density

Innovation

• Multi-modal Evidence Grounding
• Semantic Certainty Anchoring
• MEG-RAG
• Multimodal Retrieval-Augmented Generation
• Evidence Reranking