On Preserving the Knowledge of Long Clinical Texts

2023-11-02

AI Summary

Long clinical texts—such as MIMIC-III admission notes—exceed standard Transformer input limits, leading to critical medical information loss and degraded predictive performance. To address this, we propose a multi-encoder framework integrating model ensembling with segment-wise feature aggregation. This is the first approach to systematically combine ensemble learning with cross-segment semantic aggregation, effectively overcoming the fixed-length constraint of single encoders and enabling holistic modeling of full clinical narratives. On mortality prediction, our method achieves a 3.2% AUC improvement; on length-of-stay (LOS) prediction, it reduces MAE by 11.7%, consistently outperforming state-of-the-art baselines. Extensive experiments demonstrate its effectiveness, robustness, and generalizability across diverse, unstructured long clinical texts. The framework establishes a scalable new paradigm for long-context modeling in healthcare large language models.

Abstract

Clinical texts, such as admission notes, discharge summaries, and progress notes, contain rich and valuable information that can be used for clinical decision making. However, a severe bottleneck in using transformer encoders for processing clinical texts comes from the input length limit of these models: transformer-based encoders use fixed-length inputs. Therefore, these models discard part of the inputs while processing medical text. There is a risk of losing vital knowledge from clinical text if only part of it is processed. This paper proposes a novel method to preserve the knowledge of long clinical texts in the models using aggregated ensembles of transformer encoders. Previous studies used either ensemble or aggregation, but we studied the effects of fusing these methods. We trained several pre-trained BERT-like transformer encoders on two clinical outcome tasks: mortality prediction and length of stay prediction. Our method achieved better results than all baseline models for prediction tasks on long clinical notes. We conducted extensive experiments on the MIMIC-III clinical database's admission notes by combining multiple unstructured and high-dimensional datasets, demonstrating our method's effectiveness and superiority over existing approaches. This study shows that fusing ensemble and aggregation improves the model performance for clinical prediction tasks, particularly the mortality and the length of hospital stay.

Problem

• Medical Records
• Information Loss
• Decision Support

Innovation

• Combined Aggregation Strategies
• Smart Tool Utilization
• Medical Record Analysis