Systematic Comparative Analysis of Large Pretrained Language Models on Contextualized Medication Event Extraction

2025-09-23

AI Summary

This study addresses the extraction of contextual information from textual medication events in electronic health records (EHRs). Using the CMED dataset, we systematically evaluate five pre-trained language models—BERT-Base, BioBERT, Bio+Clinical BERT, RoBERTa, and Clinical Longformer—across three subtasks: drug identification, event detection, and multidimensional context classification. Results show that clinical-domain pre-trained models (e.g., Bio+Clinical BERT) significantly outperform others in drug and event detection, whereas the general-purpose BERT-Base achieves the highest F1 score in context classification, attributable to its superior semantic generalization capacity. Crucially, this work reveals, for the first time, a non-monotonic relationship between pre-training domain adaptation and task type—highlighting that domain specialization does not uniformly benefit all medical information extraction subtasks. These findings provide empirical guidance and methodological insights for model selection in clinical NLP applications.

Technology Category

• Natural Language Processing: Information Extraction
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Attention-based models have become the leading approach in modeling medical language for Natural Language Processing (NLP) in clinical notes. These models outperform traditional techniques by effectively capturing contextual rep- resentations of language. In this research a comparative analysis is done amongst pre- trained attention based models namely Bert Base, BioBert, two variations of Bio+Clinical Bert, RoBerta, and Clinical Long- former on task related to Electronic Health Record (EHR) information extraction. The tasks from Track 1 of Harvard Medical School's 2022 National Clinical NLP Challenges (n2c2) are considered for this comparison, with the Contextualized Medication Event Dataset (CMED) given for these task. CMED is a dataset of unstructured EHRs and annotated notes that contain task relevant information about the EHRs. The goal of the challenge is to develop effective solutions for extracting contextual information related to patient medication events from EHRs using data driven methods. Each pre-trained model is fine-tuned and applied on CMED to perform medication extraction, medical event detection, and multi-dimensional medication event context classification. Pro- cessing methods are also detailed for breaking down EHRs for compatibility with the applied models. Performance analysis has been carried out using a script based on constructing medical terms from the evaluation portion of CMED with metrics including recall, precision, and F1-Score. The results demonstrate that models pre-trained on clinical data are more effective in detecting medication and medication events, but Bert Base, pre- trained on general domain data showed to be the most effective for classifying the context of events related to medications.

Problem

• Extracting contextual medication event information from Electronic Health Records
• Comparing pretrained language models for medical information extraction tasks
• Developing data-driven methods for medication event detection and classification

Innovation

• Fine-tuned multiple pretrained attention-based language models
• Applied models to extract medication events from EHRs
• Compared clinical versus general domain pretrained models