MEDMKG: Benchmarking Medical Knowledge Exploitation with Multimodal Knowledge Graph

2025-05-22

AI Summary

Medical multimodal knowledge graphs (MMKGs) remain underdeveloped, hindering effective alignment between medical imaging and clinical concepts. Method: We propose MEDMKG—the first medical MMKG integrating radiographic images and clinical text—built via a multi-stage pipeline that fuses MIMIC-CXR imaging data with UMLS-structured clinical knowledge. We introduce Neighbor-aware Filtering (NaF), a novel algorithm to enhance graph quality and compactness, and jointly leverage rule-based engines and large language models for cross-modal concept extraction and relational modeling. Contribution/Results: Extensive evaluation across six datasets and three knowledge-intensive tasks demonstrates consistent performance gains over 24 baselines and four vision-language backbone models. This work establishes the first scalable, reusable MMKG framework for medicine, providing a robust benchmark and foundational infrastructure for healthcare AI.

Technology Category

• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Medical deep learning models depend heavily on domain-specific knowledge to perform well on knowledge-intensive clinical tasks. Prior work has primarily leveraged unimodal knowledge graphs, such as the Unified Medical Language System (UMLS), to enhance model performance. However, integrating multimodal medical knowledge graphs remains largely underexplored, mainly due to the lack of resources linking imaging data with clinical concepts. To address this gap, we propose MEDMKG, a Medical Multimodal Knowledge Graph that unifies visual and textual medical information through a multi-stage construction pipeline. MEDMKG fuses the rich multimodal data from MIMIC-CXR with the structured clinical knowledge from UMLS, utilizing both rule-based tools and large language models for accurate concept extraction and relationship modeling. To ensure graph quality and compactness, we introduce Neighbor-aware Filtering (NaF), a novel filtering algorithm tailored for multimodal knowledge graphs. We evaluate MEDMKG across three tasks under two experimental settings, benchmarking twenty-four baseline methods and four state-of-the-art vision-language backbones on six datasets. Results show that MEDMKG not only improves performance in downstream medical tasks but also offers a strong foundation for developing adaptive and robust strategies for multimodal knowledge integration in medical artificial intelligence.

Problem

• Integrating multimodal medical knowledge graphs is underexplored
• Lack of resources linking imaging data with clinical concepts
• Enhancing medical AI performance with unified visual-textual knowledge

Innovation

• Multimodal knowledge graph linking images and texts
• Rule-based and LLM-based concept extraction
• Neighbor-aware Filtering for graph compactness