Heartcare Suite: Multi-dimensional Understanding of ECG with Raw Multi-lead Signal Modeling

2025-06-06

AI Summary

This study addresses the lack of a unified multimodal framework for fine-grained electrocardiogram (ECG) understanding. To this end, we propose Heartcare—a comprehensive framework comprising: (1) Heartcare-220K, the first high-quality, multi-task ECG dataset covering disease diagnosis, waveform morphology analysis, and rhythm interpretation; (2) Heartcare-Bench, the first structured, multidimensional evaluation benchmark for ECG understanding; and (3) HeartcareGPT, an ECG-specialized large language model incorporating the BEAT semantic discretization tokenizer, dual-layer vector quantization, and query-guided bidirectional diffusion. Evaluated across multiple clinical tasks, Heartcare achieves state-of-the-art performance, significantly advancing multimodal ECG comprehension. Heartcare-220K and Heartcare-Bench are publicly released; HeartcareGPT demonstrates strong generalizability and clinical applicability.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

We present Heartcare Suite, a multimodal comprehensive framework for finegrained electrocardiogram (ECG) understanding. It comprises three key components: (i) Heartcare-220K, a high-quality, structured, and comprehensive multimodal ECG dataset covering essential tasks such as disease diagnosis, waveform morphology analysis, and rhythm interpretation. (ii) Heartcare-Bench, a systematic and multi-dimensional benchmark designed to evaluate diagnostic intelligence and guide the optimization of Medical Multimodal Large Language Models (Med-MLLMs) in ECG scenarios. and (iii) HeartcareGPT with a tailored tokenizer Bidirectional ECG Abstract Tokenization (Beat), which compresses raw multi-lead signals into semantically rich discrete tokens via duallevel vector quantization and query-guided bidirectional diffusion mechanism. Built upon Heartcare-220K, HeartcareGPT achieves strong generalization and SoTA performance across multiple clinically meaningful tasks. Extensive experiments demonstrate that Heartcare Suite is highly effective in advancing ECGspecific multimodal understanding and evaluation. Our project is available at https://github.com/Wznnnnn/Heartcare-Suite .

Problem

• Develops a multimodal framework for detailed ECG analysis
• Creates a high-quality dataset for ECG disease diagnosis
• Introduces a model for raw multi-lead signal compression

Innovation

• High-quality multimodal ECG dataset Heartcare-220K
• Systematic benchmark Heartcare-Bench for Med-MLLMs
• HeartcareGPT with Bidirectional ECG Abstract Tokenization