Wave2Word: A Multimodal Transformer Framework for Joint EEG-Text Alignment and Multi-Task Representation Learning in Neurocritical Care

2026-01-02

arXiv.org

AI Summary

This work addresses the limitations of existing EEG modeling approaches, which predominantly rely on discrete labels and thus fail to capture the overlapping nature of abnormal patterns, graded inter-expert agreement, temporal persistence, and alignment with clinical language. To overcome these challenges, the authors propose a multimodal Transformer framework featuring dual encoders that process bipolar montages and time-frequency representations, fused via an adaptive gating mechanism. For the first time, structured clinical text is incorporated as a supervisory signal: contrastive learning aligns EEG embeddings with expert consensus, while a novel EEG-conditioned text reconstruction loss regularizes the representation. The method achieves 0.9797 accuracy across six EEG pattern classes. Ablation studies demonstrate that removing contrastive alignment drastically reduces cross-modal retrieval Recall@10 from 0.3390 to 0.0045, confirming the efficacy of deeply coupling neural representations with clinical semantics.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Continuous electroencephalography (EEG) is routinely used in neurocritical care to monitor seizures and other harmful brain activity, including rhythmic and periodic patterns that are clinically significant. Although deep learning methods have achieved high accuracy in seizure detection, most existing approaches remain seizure-centric, rely on discrete-label supervision, and are primarily evaluated using accuracy-based metrics. A central limitation of current EEG modeling practice is the weak correspondence between learned representations and how EEG findings are interpreted and summarized in clinical workflows. Harmful EEG activity exhibits overlapping patterns, graded expert agreement, and temporal persistence, which are not well captured by classification objectives alone. This work proposes a multimodal EEG representation learning framework that integrates signal-domain modeling with structured clinical language supervision. First, raw EEG is transformed into a longitudinal bipolar montage and time-frequency representations. Second, dual transformer-based encoders model complementary temporal and frequency-centric dependencies and are fused using an adaptive gating mechanism. Third, EEG embeddings are aligned with structured expert consensus descriptions through a contrastive objective. Finally, an EEG-conditioned text reconstruction loss is introduced as a representation-level constraint alongside standard classification loss. Experimental evaluation using a controlled train-validation-test split achieves a six-class test accuracy of 0.9797. Ablation analyses show that removing contrastive alignment reduces cross-modal retrieval performance from Recall@10 of 0.3390 to 0.0045, despite minimal change in classification accuracy. These findings demonstrate that discriminative accuracy does not reliably reflect representation quality for clinically meaningful EEG modeling.

Problem

• EEG representation learning
• clinical interpretation alignment
• multimodal EEG-text modeling
• neurocritical care
• classification limitations

Innovation

• multimodal transformer
• EEG-text alignment
• contrastive learning
• representation learning
• neurocritical care