Learning from Medical Entity Trees: An Entity-Centric Medical Data Engineering Framework for MLLMs

2026-04-28

AI Summary

Existing approaches to constructing medical multimodal datasets typically partition data by modality or clinical department at a coarse granularity, failing to capture the hierarchical structure and intricate relationships inherent in clinical knowledge, thereby limiting the fine-grained comprehension and complex reasoning capabilities of multimodal large language models. To address this, this work proposes a medical-entity-centric data engineering framework that, for the first time, integrates hierarchical medical knowledge directly into the data construction pipeline. The framework automatically constructs a Medical Entity Tree (MET) from authoritative literature to guide node-driven image-text retrieval, employs a two-stage filtering and alignment process, and performs knowledge-aware data synthesis, all under structural constraints to generate high-quality visual question-answering samples. Evaluated across six medical benchmarks, this approach significantly enhances the performance of general-purpose multimodal large models, achieving state-of-the-art results on complex clinical question-answering tasks.

Abstract

Multimodal Large Language Models (MLLMs) have shown transformative potential in medical applications, yet their performance is hindered by conventional data curation strategies that rely on coarse-grained partitioning by modality or department. Such fragmented approaches fail to capture the hierarchical and interconnected nature of clinical medical knowledge, limiting the models' ability to perform fine-grained recognition and complex reasoning. In this paper, we propose a novel Entity-Centric Medical Data Engineering framework. We automatically extract entities from authoritative medical literature to construct a Medical Entity Tree (MET), a hierarchical structure that systematically encodes diseases, anatomical structures, modalities, and symptoms into a unified knowledge repository. Building upon the MET, we propose an advanced data engine that includes: (1) node-guided retrieval to anchor raw data to specific medical concepts, (2) a two-stage hybrid filtering and alignment pipeline to ensure precise visual-semantic correspondence, and (3) knowledge-aware data synthesis to generate enriched captions and targeted reasoning VQA pairs, leveraging structural constraints. Extensive evaluations across six medical benchmarks demonstrate that our approach significantly enhances the medical capabilities of general-purpose MLLMs, improving their ability to handle complex clinical queries and achieve state-of-the-art performance in diverse medical contexts.

Problem

• Multimodal Large Language Models
• medical data curation
• hierarchical medical knowledge
• fine-grained recognition
• complex reasoning

Innovation

• Medical Entity Tree
• Entity-Centric Data Engineering
• Multimodal Large Language Models
• Knowledge-Aware Data Synthesis
• Visual-Semantic Alignment