Position: Evaluation of ECG Representations Must Be Fixed

2026-02-19

AI Summary

Current evaluation of 12-lead ECG representation learning overly emphasizes arrhythmia classification and waveform morphology, neglecting critical clinical tasks such as structural heart disease, hemodynamic status, and patient prognosis, thereby creating a gap between research and real-world clinical needs. This work proposes a multitask evaluation framework that integrates datasets including PTB-XL, CPSC2018, and CSN, augmented with newly curated clinical labels, to systematically benchmark existing methods under class-imbalanced and multi-label settings. The study reveals for the first time the limitations of mainstream pretrained models on extended clinical tasks, showing that randomly initialized encoders can match or even surpass advanced models under linear probing. Crucially, adopting more rigorous evaluation protocols substantially alters performance rankings, underscoring the urgent need for reform in representation learning assessment standards for ECG analysis.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Computer Vision: Representation Learning for Vision
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

This position paper argues that current benchmarking practice in 12-lead ECG representation learning must be fixed to ensure progress is reliable and aligned with clinically meaningful objectives. The field has largely converged on three public multi-label benchmarks (PTB-XL, CPSC2018, CSN) dominated by arrhythmia and waveform-morphology labels, even though the ECG is known to encode substantially broader clinical information. We argue that downstream evaluation should expand to include an assessment of structural heart disease and patient-level forecasting, in addition to other evolving ECG-related endpoints, as relevant clinical targets. Next, we outline evaluation best practices for multi-label, imbalanced settings, and show that when they are applied, the literature's current conclusion about which representations perform best is altered. Furthermore, we demonstrate the surprising result that a randomly initialized encoder with linear evaluation matches state-of-the-art pre-training on many tasks. This motivates the use of a random encoder as a reasonable baseline model. We substantiate our observations with an empirical evaluation of three representative ECG pre-training approaches across six evaluation settings: the three standard benchmarks, a structural disease dataset, hemodynamic inference, and patient forecasting.

Problem

• ECG representation learning
• benchmarking
• clinical evaluation
• structural heart disease
• multi-label classification

Innovation

• ECG representation learning
• evaluation benchmark
• clinical relevance
• random encoder baseline
• multi-label imbalance