MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generation

2026-06-24

AI Summary

Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation (MKG-RAG) faces significant challenges, including the difficulty of aligning heterogeneous multimodal knowledge and the inadequacy of existing retrievers. This work proposes MKG-RAG-Bench, the first cross-domain benchmark specifically designed for MKG-RAG, which leverages large language models for automated curation to construct structured question-answering datasets from multimodal knowledge graphs in both general and medical domains. Notably, it treats retrieval as a core evaluation objective, enables controlled assessment under diverse modality configurations, and provides precise supervisory signals to facilitate inefficient knowledge filtering and structured query generation. Experimental results demonstrate that multimodal retrieval quality substantially impacts end-to-end generation performance, revealing current system bottlenecks and establishing a foundational framework for future MKG-RAG research.

Abstract

Retrieval-augmented generation (RAG) over knowledge graphs has emerged as a promising approach for grounding large language models, yet existing benchmarks largely overlook the challenges of retrieval in multimodal knowledge graph RAG (MKG-RAG). In practice, retrieval is a critical bottleneck: multimodal knowledge is heterogeneous, difficult to align across modalities, and often poorly served by retrievers designed for unstructured corpora. To address this gap, we introduce MKG-RAG-Bench, a cross-domain benchmark explicitly designed to evaluate retrieval in MKG-RAG. MKG-RAG-Bench is constructed from two multimodal knowledge graphs spanning general and medical domains, and includes carefully aligned question-answering datasets that support controlled evaluation of both retrieval and downstream generation. The benchmark is built using an LLM-based curation pipeline that filters low-utility knowledge, generates structurally grounded queries with exact supervision, and systematically covers diverse modality configurations. Through extensive experiments across representative retriever families and modality settings, we show that effective multimodal retrieval remains challenging yet crucial for end-to-end MKG-RAG performance, and that retrieval quality strongly determines generation outcomes. By isolating retrieval as a first-class evaluation target, MKG-RAG-Bench provides a principled foundation for diagnosing current limitations and advancing multimodal knowledge graph RAG systems.

Problem

• multimodal knowledge graph
• retrieval-augmented generation
• retrieval bottleneck
• cross-modal alignment
• heterogeneous knowledge

Innovation

• multimodal knowledge graph
• retrieval-augmented generation
• benchmark
• cross-modal alignment
• LLM-based curation