Towards Visualizing Electronic Medical Records via Natural Language Queries

2025-06-15

AI Summary

Electronic medical records (EMRs) contain heterogeneous structured and unstructured data, posing significant challenges for visual modeling; moreover, high-quality annotated datasets for medical visualization are scarce and prohibitively expensive to construct. Method: This paper proposes a zero-annotation, large language model–driven paradigm for synthetic visual data generation, introducing MedicalVis—the first large-scale EMR text-to-visualization benchmark dataset comprising 35,374 diverse (text, chart) pairs. We further release a dedicated evaluation framework and present MedCodeT5, a specialized model fine-tuned for natural language–driven statistical chart generation. Results: MedCodeT5 substantially outperforms existing baselines on this task. MedicalVis establishes a standardized, open-source benchmark for medical visualization research, enabling rigorous evaluation and fostering the deployment of interpretable AI and clinical decision support systems in real-world healthcare settings.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Web data visualization

Abstract

Electronic medical records (EMRs) contain essential data for patient care and clinical research. With the diversity of structured and unstructured data in EHR, data visualization is an invaluable tool for managing and explaining these complexities. However, the scarcity of relevant medical visualization data and the high cost of manual annotation required to develop such datasets pose significant challenges to advancing medical visualization techniques. To address this issue, we propose an innovative approach using large language models (LLMs) for generating visualization data without labor-intensive manual annotation. We introduce a new pipeline for building text-to-visualization benchmarks suitable for EMRs, enabling users to visualize EMR statistics through natural language queries (NLQs). The dataset presented in this paper primarily consists of paired text medical records, NLQs, and corresponding visualizations, forming the first large-scale text-to-visual dataset for electronic medical record information called MedicalVis with 35,374 examples. Additionally, we introduce an LLM-based approach called MedCodeT5, showcasing its viability in generating EMR visualizations from NLQs, outperforming various strong text-to-visualization baselines. Our work facilitates standardized evaluation of EMR visualization methods while providing researchers with tools to advance this influential field of application. In a nutshell, this study and dataset have the potential to promote advancements in eliciting medical insights through visualization.

Problem

• Visualizing EMRs via natural language queries efficiently
• Overcoming manual annotation costs for medical visualization data
• Generating EMR visualizations using LLMs without intensive labor

Innovation

• LLMs generate EMR visualization data automatically
• Pipeline creates text-to-visual benchmarks for EMRs
• MedCodeT5 model excels in NLQ-based visualization