A novel language model for predicting serious adverse event results in clinical trials from their prospective registrations

2025-07-21

AI Summary

This study addresses the challenge of predicting severe adverse event (SAE) incidence risk from lengthy prospective clinical trial registration texts to support trial design optimization and dynamic safety monitoring. To overcome input-length limitations of pretrained language models (e.g., ClinicalT5, BioBERT), we propose a sliding-window embedding strategy that preserves semantic integrity in long documents. We further introduce the first systematic framework for quantitative SAE rate prediction based solely on registration data—including binary classification of high vs. low SAE rates, treatment-control group differentiation, and regression of SAE proportions. Experimental results show that the best-performing model achieves an AUC of 77.6% for SAE rate classification and an RMSE of 18.6% for control-group SAE proportion prediction. The sliding-window approach improves average AUC by 2.00% and reduces average RMSE by 1.58% across 12 models, significantly enhancing long-text modeling capability and prediction robustness.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Calibration & Uncertainty Quantification
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Objectives: With accurate estimates of expected safety results, clinical trials could be designed to avoid terminations and limit exposing participants to unnecessary risks. We evaluated methods for predicting serious adverse event (SAE) results in clinical trials using information only from their registrations prior to the trial. Material and Methods: We analysed 22,107 two-arm parallel interventional clinical trials from ClinicalTrials.gov with structured summary results. Two prediction models were developed: a classifier predicting will experimental arm have higher SAE rates (area under the receiver operating characteristic curve; AUC) than control arm, and a regression model to predict the proportion of SAEs in control arms (root mean squared error; RMSE). A transfer learning approach using pretrained language models (e.g., ClinicalT5, BioBERT) was used for feature extraction, combined with downstream model for prediction. To maintain semantic representation in long trial texts exceeding localised language model input limits, a sliding window method was developed for embedding extraction. Results: The best model (ClinicalT5+Transformer+MLP) had 77.6% AUC predicting which trial arm has a higher proportion of patients with SAEs. When predicting proportion of participants experiencing SAE in the control arm, the same model achieved RMSE of 18.6%. The sliding window approach consistently outperformed methods without it. Across 12 classifiers, the average absolute AUC increase was 2.00%; across 12 regressors, the average absolute RMSE reduction was 1.58%. Discussion: Summary results data available at ClinicalTrials.gov remains underutilised. The potential to estimate results of trials before they start is an opportunity to improve trial design and flag discrepancies between expected and reported safety results.

Problem

• Predicts serious adverse event results in clinical trials
• Uses trial registration data for safety outcome estimation
• Develops models for SAE proportion comparison and prediction

Innovation

• Transfer learning with pretrained clinical language models
• Sliding window method for long text embedding
• Combined classifier and regression model approach