ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation

2026-02-04

AI Summary

This work addresses the critical issue of clinically unreliable hallucinations commonly exhibited by existing multimodal large language models (MLLMs) in electrocardiogram (ECG) interpretation. To enhance diagnostic reliability, we propose the first reasoning-oriented MLLM specifically designed for trustworthy ECG analysis. Our approach integrates protocol-guided instruction data generation, a decoupled architecture with interleaved modality dropout, and a reinforcement learning mechanism grounded in diagnostic evidence. These innovations collectively improve both model accuracy and cross-modal consistency. Notably, this study provides the first quantitative demonstration of the pervasive and severe hallucination problem in current models. To advance trustworthy multimodal medical AI, we publicly release our code and an online platform for community use and further research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Electrocardiography (ECG) serves as an indispensable diagnostic tool in clinical practice, yet existing multimodal large language models (MLLMs) remain unreliable for ECG interpretation, often producing plausible but clinically incorrect analyses. To address this, we propose ECG-R1, the first reasoning MLLM designed for reliable ECG interpretation via three innovations. First, we construct the interpretation corpus using \textit{Protocol-Guided Instruction Data Generation}, grounding interpretation in measurable ECG features and monograph-defined quantitative thresholds and diagnostic logic. Second, we present a modality-decoupled architecture with \textit{Interleaved Modality Dropout} to improve robustness and cross-modal consistency when either the ECG signal or ECG image is missing. Third, we present \textit{Reinforcement Learning with ECG Diagnostic Evidence Rewards} to strengthen evidence-grounded ECG interpretation. Additionally, we systematically evaluate the ECG interpretation capabilities of proprietary, open-source, and medical MLLMs, and provide the first quantitative evidence that severe hallucinations are widespread, suggesting that the public should not directly trust these outputs without independent verification. Code and data are publicly available at \href{https://github.com/PKUDigitalHealth/ECG-R1}{here}, and an online platform can be accessed at \href{http://ai.heartvoice.com.cn/ECG-R1/}{here}.

Problem

• ECG interpretation
• multimodal large language models
• clinical hallucination
• diagnostic reliability
• modality-agnostic

Innovation

• Protocol-Guided Instruction Data Generation
• Modality-Decoupled Architecture
• Interleaved Modality Dropout
• Reinforcement Learning with Diagnostic Evidence Rewards
• ECG Interpretation Reliability