VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation

2025-06-11

arXiv.org

AI Summary

Existing multimodal knowledge graphs (MMKGs) suffer from narrow knowledge coverage, outdated updates, and limited modality support—primarily restricted to text and images—thereby hindering their generalization in cross-modal tasks. To address these limitations, we propose VAT-KG, the first vision-audio-text tri-modal, concept-centric knowledge graph, where each triple is grounded in raw multimodal data and annotated with fine-grained semantic descriptions. Methodologically, we introduce: (1) the first concept-level multimodal knowledge alignment framework; (2) a generalizable, automated MMKG construction pipeline; and (3) a fine-grained retrieval-augmented generation (RAG) framework supporting arbitrary-modal queries. Experiments demonstrate that VAT-KG significantly enhances multimodal large language models’ (MLLMs) performance on multimodal question answering. It achieves substantial advances in knowledge breadth, temporal freshness, and modality completeness—setting a new benchmark for tri-modal knowledge representation and reasoning.

Technology Category

• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multimodal Knowledge Graphs (MMKGs), which represent explicit knowledge across multiple modalities, play a pivotal role by complementing the implicit knowledge of Multimodal Large Language Models (MLLMs) and enabling more grounded reasoning via Retrieval Augmented Generation (RAG). However, existing MMKGs are generally limited in scope: they are often constructed by augmenting pre-existing knowledge graphs, which restricts their knowledge, resulting in outdated or incomplete knowledge coverage, and they often support only a narrow range of modalities, such as text and visual information. These limitations reduce their extensibility and applicability to a broad range of multimodal tasks, particularly as the field shifts toward richer modalities such as video and audio in recent MLLMs. Therefore, we propose the Visual-Audio-Text Knowledge Graph (VAT-KG), the first concept-centric and knowledge-intensive multimodal knowledge graph that covers visual, audio, and text information, where each triplet is linked to multimodal data and enriched with detailed descriptions of concepts. Specifically, our construction pipeline ensures cross-modal knowledge alignment between multimodal data and fine-grained semantics through a series of stringent filtering and alignment steps, enabling the automatic generation of MMKGs from any multimodal dataset. We further introduce a novel multimodal RAG framework that retrieves detailed concept-level knowledge in response to queries from arbitrary modalities. Experiments on question answering tasks across various modalities demonstrate the effectiveness of VAT-KG in supporting MLLMs, highlighting its practical value in unifying and leveraging multimodal knowledge.

Problem

• Addressing outdated knowledge coverage in multimodal knowledge graphs
• Overcoming modality limitations in existing multimodal knowledge graphs
• Enhancing multimodal reasoning through concept-centric knowledge alignment

Innovation

• First concept-centric multimodal knowledge graph
• Automatic generation pipeline with cross-modal alignment
• Novel multimodal retrieval framework for arbitrary queries