MedPix 2.0: A Comprehensive Multimodal Biomedical Dataset for Advanced AI Applications

2024-07-03

arXiv.org

AI Summary

A high-quality, privacy-compliant, multimodal biomedical dataset enabling synergistic integration of heterogeneous clinical data is currently lacking, hindering the advancement of medical AI. Method: We introduce MedPix 2.0—the first open-source multimodal dataset jointly comprising CT/MRI images and structured clinical reports—curated via a semi-automated cleaning pipeline followed by rigorous human validation to ensure data quality and regulatory compliance (e.g., HIPAA/GDPR). We further propose DR-Minerva+KG, the first framework integrating knowledge graphs (KGs) with retrieval-augmented generation (RAG) in a unified multimodal large language model (MLLM), enabling end-to-end medical reasoning and natural-language question answering. The system features an interactive database interface built on MongoDB and a Python GUI, incorporating Llama 3.1 Instruct 8B, cross-modal alignment, and fine-grained annotation. Results: DR-Minerva achieves >92% accuracy on anatomical localization and imaging modality identification; KG augmentation significantly improves clinical QA accuracy.

Technology Category

• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The increasing interest in developing Artificial Intelligence applications in the medical domain, suffers from the lack of high-quality data set, mainly due to privacy-related issues. In addition, the recent increase in large multimodal models (LMM) leads to the need for multimodal medical data sets, where clinical reports and findings are attached to the corresponding CT or MRI scans. This paper illustrates the entire workflow for building the MedPix 2.0 data set. Starting with the well-known multimodal data set MedPix extsuperscript{ extregistered}, mainly used by physicians, nurses, and healthcare students for Continuing Medical Education purposes, a semi-automatic pipeline was developed to extract visual and textual data followed by a manual curing procedure in which noisy samples were removed, thus creating a MongoDB database. Along with the data set, we developed a GUI aimed at navigating efficiently the MongoDB instance and obtaining the raw data that can be easily used for training and/or fine-tuning LMMs. To enforce this point, in this work, we first recall DR-Minerva, a RAG-based LMM trained using MedPix 2.0. DR-Minerva predicts the body part and the modality used to scan its input image. We also propose the extension of DR-Minerva with a Knowledge Graph that uses Llama 3.1 Instruct 8B, and leverages MedPix 2.0. The resulting architecture can be queried in a end-to-end manner, as a medical decision support system. MedPix 2.0 is available on GitHub. url{https://github.com/CHILab1/MedPix-2.0}

Problem

• Biomedical Information Package
• Patient Privacy
• AI Training Data

Innovation

• MedPix 2.0
• AI training
• DR-Minerva