RelCAT: Advancing Extraction of Clinical Inter-Entity Relationships from Unstructured Electronic Health Records

2025-01-27

AI Summary

To address the challenges of implicit, fragmented, and hard-to-model clinical relationships in electronic health records (EHRs), this paper introduces RelCAT—a comprehensive toolkit for interactive annotation, modeling, and evaluation of clinical relation extraction. Methodologically, it proposes the first unified framework integrating concept recognition and relation classification; it synergistically combines CogStack MedCAT with transformer-based models (e.g., BERT, Llama), customizes annotation interfaces via MedCATTrainer, and employs transfer learning coupled with contrastive-driven fine-tuning. Contributions include: (1) the first high-robustness clinical relation extraction system validated on real-world NHS EHR data; and (2) state-of-the-art performance—achieving macro-F1 = 0.977 on the n2c2 gold-standard benchmark and F1 ≥ 0.93 on a large-scale NHS clinical dataset—demonstrating substantial improvements in generalizability and clinical applicability.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Natural Language Processing: Information Extraction

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

This study introduces RelCAT (Relation Concept Annotation Toolkit), an interactive tool, library, and workflow designed to classify relations between entities extracted from clinical narratives. Building upon the CogStack MedCAT framework, RelCAT addresses the challenge of capturing complete clinical relations dispersed within text. The toolkit implements state-of-the-art machine learning models such as BERT and Llama along with proven evaluation and training methods. We demonstrate a dataset annotation tool (built within MedCATTrainer), model training, and evaluate our methodology on both openly available gold-standard and real-world UK National Health Service (NHS) hospital clinical datasets. We perform extensive experimentation and a comparative analysis of the various publicly available models with varied approaches selected for model fine-tuning. Finally, we achieve macro F1-scores of 0.977 on the gold-standard n2c2, surpassing the previous state-of-the-art performance, and achieve performance of>=0.93 F1 on our NHS gathered datasets.

Problem

• Medical Record Analysis
• Patient Information Correlation
• Efficient Data Understanding

Innovation

• RelCAT
• machine learning models
• medical entity relationship extraction